Function Call ABI: Register Assignment on x86-64

A function call is a contract about where values live, and the contract is not in the source. The source says int f(int, int, int); the machine needs to know that the first argument arrives in edi, the second in esi, the third in edx, that the return value comes back in eax, and that the callee may destroy rcx but must restore rbx. Every one of those decisions is fixed by the ABI, and both the caller and the callee are compiled against it independently.

The System V AMD64 ABI, used by Linux, the BSDs, macOS, and Solaris on x86-64, is the reference case. Its argument-passing rules are short enough to state fully, but the structure-passing rules that sit on top of them produce the surprises.

The register file as the contract

The ABI divides the general-purpose registers into two classes. Caller-saved registers (rax, rcx, rdx, rsi, rdi, r8 through r11) may be overwritten by the callee without warning; the caller that needs a value across a call must spill it to memory first. Callee-saved registers (rbx, rbp, r12 through r15) must be restored to their incoming values before the callee returns. The stack pointer rsp is restored implicitly by the epilogue.

Six general-purpose registers carry integer arguments, in order: rdi, rsi, rdx, rcx, r8, r9. Eight vector registers carry floating-point and SIMD arguments: xmm0 through xmm7. Arguments are assigned in declaration order from these two independent pools, so a call with three integers and three doubles uses rdi, rsi, rdx for the integers and xmm0, xmm1, xmm2 for the doubles. When a pool runs out, the remaining arguments go on the stack.

Returns use a smaller set. An integer or pointer result comes back in rax, a second eightbyte of an integer result in rdx, a floating-point result in xmm0, and a second eightbyte of a floating-point result in xmm1. A caller that expects a 128-bit integer result reads both rax and rdx; a caller that expects a double reads only xmm0 and ignores rax.

Two further details are easy to miss. The direction flag in rflags must be clear on entry and on return, so a callee that uses string instructions must clear it. And the x87/MMX state is not part of the argument convention at all; the ABI specifies that the x87 stack must be empty on entry to a function, so a callee that pushes onto it must pop before returning.

Sixteen bytes, one eightbyte at a time

A scalar argument maps to one register slot. A structure does not, and the ABI handles it by a classification procedure rather than by a simple size test. The argument is divided into consecutive eight-byte chunks, each called an eightbyte, and each chunk receives a class: INTEGER, SSE, SSEUP, X87, X87UP, COMPLEX_X87, MEMORY, or NO_CLASS for padding.

The rules for a plain aggregate work in two stages. First, an aggregate that exceeds the size the ABI allows in registers, or that contains a field whose alignment is not satisfied by its offset, is classified MEMORY. Otherwise each eightbyte is classified from the fields that overlap it: integer and pointer fields contribute INTEGER, float and double contribute SSE, and a chunk with no fields is NO_CLASS and is ignored. When two fields of different classes share one eightbyte, INTEGER wins, so a chunk holding a float and an int is passed as an integer.

Second, a post-merger applies across the chunks. Any MEMORY chunk makes the whole argument memory, and an aggregate longer than two eightbytes is passed in memory unless every chunk is SSE or SSEUP. That last rule is what allows four float values to travel in two xmm registers while three long values do not travel in registers at all.

struct pair_f  { float a, b; };             /* one eightbyte, SSE  -> xmm0     */
struct mixed   { float a, b, c; int tag; }; /* SSE, then INTEGER   -> xmm0, rdi */
struct wide    { long a, b, c; };           /* 24 bytes, not SSE   -> memory   */
struct odd     { char c; double d; };       /* INTEGER, then SSE   -> rdi, xmm0 */

The second case is the one that catches people. struct mixed is 16 bytes and its last two fields, c and tag, share the second eightbyte, so that whole chunk becomes INTEGER and the float is copied bit-for-bit into a general-purpose register. The callee recovers it with a move back into an xmm register. A structure of four float values, by contrast, keeps both chunks SSE and needs no such move.

The fourth case shows the opposite arrangement: the chunks are classified independently, so a structure can consume one integer register and one vector register at the same time. The registers are allocated from the two pools in declaration order, which means an aggregate that mixes classes can exhaust the integer pool before a later scalar argument is placed.

The case that genuinely forces memory is an unaligned member, which arises with packed structures or with a member placed at an offset that violates its alignment. The classification cannot express “this half is a double at an odd offset”, so the ABI’s fallback is the stack, and the caller must store the whole aggregate into the outgoing argument area before the call.

Why the stack is still 16-byte aligned

On entry to a function, rsp + 8 is a multiple of 16. The + 8 accounts for the return address that the call instruction pushed. The reason is that the ABI requires 16-byte stack alignment at every call site, and the return address occupies eight of those bytes inside the callee. A function that calls another must therefore arrange its own frame so that the outgoing rsp is 16-byte aligned, which usually means subtracting a multiple of 16 after saving registers and then adjusting for any odd number of pushed registers.

The alignment matters because SSE instructions that operate on 16-byte memory operands require it, and because some compilers emit aligned moves for spilled vector registers. Code that violates the rule may work for a long time and then fault when a particular path spills a __m128 with an aligned store.

The 128 bytes below rsp form the red zone. A leaf function, one that makes no calls, may use that region for temporaries without adjusting the stack pointer, because the kernel builds a signal frame below the red zone rather than on top of it. A function that calls another may not use the red zone at all, since the call would overwrite it.

Stack arguments are placed in increasing address order from the lowest address, with the first stack argument at [rsp + 8] after the return address. The total argument area must be a multiple of eight, and the ABI requires the end of the argument area to be aligned on a 16-byte boundary so that a vector argument placed on the stack starts at a legal address for an aligned move.

The prologue and the frame pointer

A typical non-leaf function begins with push rbp; mov rbp, rsp; sub rsp, N, saving the caller’s frame pointer and establishing its own. The frame pointer is not required; with -fomit-frame-pointer, which is the default at most optimisation levels, the compiler addresses locals relative to rsp and recomputes offsets as the stack pointer changes. The cost is that unwinding becomes harder, which is why the ABI defines call frame information separately from the frame pointer. The article on stack unwinding and call frame information covers that mechanism.

Callee-saved registers that the function uses are pushed in the prologue and popped in the epilogue. The order and the exact set are the compiler’s choice, but the total must restore rsp exactly, because the return address sits immediately above the incoming rsp and the ret instruction pops it from there.

Comparing against the other major 64-bit convention

The Arm 64-bit ABI, AAPCS64, differs in ways that matter for hand-written assembly and for cross-platform structure definitions. It provides eight integer argument registers, x0 through x7, and eight vector registers, v0 through v7, giving 16 slots against x86-64’s 14. It reserves x8 as the indirect result register, used when a function returns a structure too large to fit in registers: the caller passes the address of the destination in x8 and the callee writes there. On x86-64 the same situation uses a hidden first argument in rdi, which shifts every subsequent argument by one register.

AAPCS64 also requires 16-byte stack alignment, but it has no red zone, so a leaf function must allocate before using stack space. Structures of 16 bytes or less are passed in registers under rules that are similar in spirit but not identical in detail; the eightbyte merging rule has no direct equivalent, and the classification is specified per member type rather than by chunk.

None of these differences is visible in portable C, and all of them are visible the moment a declaration is wrong. A function compiled with a prototype that disagrees with the definition (a missing parameter, a float where the definition expects double, a structure passed by value where the definition expects a pointer) produces code that reads the wrong register. The result is not a diagnostic at the boundary; it is a value that is silently wrong, or a crash at an address that came from a register holding something else entirely. The promotion rules that make float become double in an unprototyped call are treated in variadic calls and default argument promotions.