Stack Frame Layout and Why a Linear Overflow Reaches the Return Address

A function call on x86-64 is a contract between two pieces of code that never see each other’s source. The caller puts the first six integer arguments in registers, the call instruction pushes an eight-byte return address, and the callee promises to restore every register it clobbers before returning. The frame is where the callee keeps the bookkeeping for that promise. Its layout is chosen by the compiler under rules fixed by the ABI, not by the order of declarations in the source, and that fixed layout is what turns a linear write past the end of a local array into control over where the program goes next.

The direction the frame grows

The stack grows toward lower addresses. push subtracts eight from rsp and stores; sub rsp, 0x40 makes room for locals; pop reads and adds eight back. Locals therefore sit at negative displacements from the frame pointer, while everything the caller arranged before the call sits above it.

For a function whose frame is anchored at rbp, the layout is deterministic:

 higher addresses
   ...caller's frame...
   stack-passed arguments (7th and later)   [rbp + 16]
   return address                            [rbp + 8]
   saved rbp                                 [rbp]
   canary, when the compiler emits one       [rbp - 8]
   other locals
   char buffer[32]                           [rbp - 0x30]
 lower addresses

A buffer occupying [rbp - 0x30, rbp - 0x11] is followed in memory by the canary slot, then the saved frame pointer, then the return address. Array indices increase with address, so buffer[32] is the first byte of whatever the compiler placed next. A write of 40 bytes into a 32-byte array therefore covers eight bytes of the neighbouring slot, and a write of 56 bytes reaches the return address itself. The overflow does not need to be large, only long enough, and the distance is a compile-time constant that a reader of the disassembly can recover exactly.

The 128-byte red zone below rsp complicates the picture slightly for leaf functions. A leaf function may keep locals at negative offsets from rsp without adjusting the stack pointer at all, on the guarantee that no signal handler will clobber that region. Such a frame has no rbp chain, but the buffer still sits below the return address in memory, so the arithmetic of an overflow is unchanged.

What sits between a buffer and the return address

Source order does not determine frame order. Compilers pack locals to satisfy alignment, group variables by size to reduce padding, and freely move arrays relative to scalars. Two adjacent character arrays in the source may end up adjacent in the frame or separated by a saved register, and a compiler is entitled to reorder them between builds or between optimisation levels. An overflow written against one binary’s layout may miss entirely against the next.

The classic hazard is a pair of arrays declared together and treated as if the second were a bound on the first. A function that reads into char user[32] and then copies into char name[32] with no length check has two independent overflows, and the second one writes into a region the compiler may have placed anywhere. The reliable way to establish the distance is to read the function’s prologue: the sub rsp, N and the lea/mov offsets that address each local give the frame map directly.

Arguments beyond the sixth integer argument are also reachable. The System V ABI passes them on the stack, above the return address, so an overflow that continues past [rbp + 8] begins overwriting the caller’s copy of its own arguments. Those values are read by the caller after the callee returns, which makes them a target for corrupting a length or a pointer that the caller trusts.

Why the saved frame pointer is a target in its own right

leave is shorthand for mov rsp, rbp followed by pop rbp. It restores the stack pointer from the frame pointer and then pops the saved value of the caller’s frame pointer. If the overflow has replaced the saved rbp with an attacker-chosen value, leave installs that value as the caller’s frame pointer, and the caller’s subsequent accesses to its own locals are computed from it.

This is the frame pointer overwrite. Its value is that it does not require reaching the return address, and on 32-bit x86 it required only a single byte: the saved ebp and the return address are adjacent, and the least significant byte of ebp controls a region of the caller’s frame that can be aimed at a value the caller later uses as an index or a pointer. Because the modified value is consumed by ordinary code rather than by ret, a corrupted frame pointer can produce an effect that no return-address check would notice.

Frame pointer omission changes the arithmetic without removing the danger. At -O2 on x86-64, rbp is an ordinary callee-saved register, and the epilogue is pop rbp; ret with the stack pointer adjusted by add rsp, N. The saved rbp slot may not exist at all, and locals are addressed from rsp, so the distances shift with every push and pop inside the function. The return address is still at a fixed offset from the entry rsp, which is what an overflow actually needs.

The return address and the canonical-address constraint

ret pops eight bytes into rip. On x86-64 those eight bytes must form a canonical address: bits 63 through 48 must all equal bit 47. User-space text and library mappings live in the low half, so a valid target looks like 0x00007f... or 0x000055.... An overflow that writes eight arbitrary bytes almost always produces a non-canonical value, and the ret raises a general protection fault before any instruction at the target executes. Naive full overwrites fail for this reason rather than because of any deliberate defence.

Practical overwrites are therefore partial. The low twelve bits of an address are the page offset and are never randomised, so a two-byte overwrite reaches anywhere within the same 64 KiB region. A three-byte overwrite reaches within 16 MiB. Because x86 is little-endian, the first byte written past the buffer becomes the least significant byte of the target, so a one-byte overflow shifts the return address within a 256-byte window at a time when the high bytes are still unknown. This is the same property that makes address-space randomisation survivable for the attacker in some cases and fatal in others.

A second constraint comes from the value that terminates the write. strcpy, sprintf, and gets stop at a NUL byte and always append one, so they can never place a zero byte in the middle of an address. On a little-endian target that limits a string-driven overwrite to targets whose low bytes contain no zero, and the terminating NUL itself lands one byte past whatever was copied. memcpy, read, and recv place arbitrary bytes including NUL, which is why the choice of copying function changes the exploitability of an otherwise identical bug.

What the write primitive actually allows

Not every overflow is a linear one. A negative length converted to size_t, an int multiplication that wraps before being passed to malloc, or a signed comparison that accepts a negative count all produce writes that land far from the buffer, sometimes at an address the input controls. Those are arbitrary-write primitives and they compose with completely different techniques than the contiguous overflow described above.

The linear case is bounded by what follows the buffer. Reaching the return address requires writing through the canary slot, and the compiler’s guard is checked immediately before the epilogue, which is why stack canaries are placed exactly there rather than anywhere else in the frame. Reaching it also requires that the target be executable or that the program be steered into code that already exists, the constraint that non-executable memory imposed on every exploit written after it became default. The frame layout is the same in both cases; only the payload changes.

What makes the layout exploitable at all is that the return address is data living in a region the program writes to through pointers that carry no bounds. A language runtime that checks array bounds removes the write; a compiler that reorders the frame makes the distance uncertain but not absent; a hardware feature that keeps return addresses in a second stack removes the target. Each of those is a different answer to the same structural problem, and each costs something the plain frame does not pay.