The C Calling Convention as an Interoperability Contract

Two languages meeting at a function call are not meeting as strangers. They run on the same processor, share the same stack pointer, and are linked by the same loader. What separates them is a convention: an agreement about which registers carry which argument, who is responsible for saving state, and how the stack is aligned at the moment control transfers. Every foreign function interface is, underneath its marshalling layer, an attempt to satisfy that convention exactly.

The convention is defined by the platform ABI rather than by any language. On x86-64 Linux, the BSDs, and macOS, the governing document is the System V AMD64 psABI. On Windows the same architecture uses a different convention, and on 64-bit Arm the AAPCS64 applies everywhere. A single program that calls a native library must satisfy whichever of these its target uses, and the differences are not cosmetic.

Where the arguments actually go

Under the System V AMD64 convention the first six arguments that fit in a general-purpose register travel in rdi, rsi, rdx, rcx, r8, and r9, in that order. Floating-point arguments use a separate sequence, xmm0 through xmm7. The two sequences advance independently, so a call like f(1, 2.0, 3, 4.0) puts 1 in rdi, 2.0 in xmm0, 3 in rsi, and 4.0 in xmm1. Once either sequence is exhausted, further arguments of that class go on the stack in order, at increasing addresses.

The return value follows the same split. An integer or pointer result comes back in rax, a 128-bit integer uses rax and rdx, and a floating-point result uses xmm0. A struct too large for the register classes is returned through a hidden pointer passed as an implicit first argument, which is why a caller that ignores the size of the return type can silently corrupt rdi and shift every subsequent argument by one register.

Windows x64 uses four registers instead of six, rcx, rdx, r8, r9, and requires the caller to reserve thirty-two bytes of shadow space above the return address. The shadow space exists so the callee can spill its register arguments without adjusting the stack pointer, which makes unwinding simpler. A caller that omits it does not fail at the call site; it fails later, when the callee writes into what it believes is its own frame and destroys the caller’s locals. This is the characteristic failure of porting hand-written assembly or a hand-rolled trampoline between platforms: the code assembles, links, and runs, and corrupts memory only when the callee happens to spill.

AAPCS64 uses x0 through x7 for integers and v0 through v7 for floating point, with results in x0 and v0. It has no shadow space and no red zone. The variety is the point: no single register assignment is universal, and extern "C" does not name one.

The stack alignment invariant

The convention that breaks most often in generated code is alignment. On x86-64 System V, the stack pointer must be a multiple of sixteen at the instruction immediately before a call. Since call pushes an eight-byte return address, the callee begins execution with rsp congruent to eight modulo sixteen. A leaf function may use the 128 bytes below rsp without adjusting it, the red zone, but any function that calls another must first bring rsp back to a multiple of sixteen.

The consequence is that a hand-written trampoline which saves registers by pushing them must count the pushes. Pushing an odd number of eight-byte values leaves the stack misaligned by eight bytes, and the next call into code that uses aligned SSE moves on stack slots will fault. The fault appears in the callee, often in a library the caller does not own, which is why the diagnosis is usually wrong the first time.

AAPCS64 requires sixteen-byte alignment of sp at all times when it is used as a base for memory access, and the platform requires it at public interfaces. x86-64 Windows requires sixteen-byte alignment at the call site as well, plus the shadow space. Alignment is therefore the one part of the contract that a caller can verify mechanically, and it is the part most often violated by code that was correct on one target.

Who saves what

Registers divide into caller-saved and callee-saved sets. On System V AMD64, rbx, rbp, r12 through r15, and rsp are callee-saved: a function that uses them must restore their incoming values before returning. Everything else, including all of xmm0–xmm15 and the argument registers, may be destroyed by any call.

This split is the reason a language runtime cannot keep a live value in a caller-saved register across a call into foreign code. If the runtime’s compiled code assumes that r10 holds a heap base pointer, a call to a C function is free to overwrite r10, and the runtime’s assumption becomes false at exactly the point where it stops being checkable. Runtimes that call out therefore either treat the call as clobbering everything outside the callee-saved set, or spill their pinned values to a frame the callee cannot reach.

Two further requirements are easy to miss because they concern state rather than registers. The direction flag must be clear on entry to a function and on return, because C code assumes string operations run forward; a foreign routine that returns with the direction flag set will make the next memcpy in the caller walk backwards. And on System V, a variadic call sets al to the number of vector registers used, so the callee knows how much of the SSE save area to spill. A hand-written call into printf that leaves al unset works until the first floating-point argument is passed.

What extern C promises, and what it does not

In C++, extern "C" is a linkage specification. It changes the symbol name the compiler emits and disables C++-specific argument typing rules; it does not select a calling convention. On most 64-bit platforms there is only one convention in use, so the distinction is invisible. On 32-bit x86 Windows, cdecl and stdcall differ in who pops the arguments and in the decorated symbol name, and extern "C" alone does not choose between them; the compiler’s default applies. Code that passes a callback pointer across such a boundary must match the convention the receiving side expects, and the type system will not catch the mismatch.

extern "C" also says nothing about struct layout, enum size, bitfield packing, or the signedness of char. Those are properties of the platform’s C ABI, and a language that models its own records differently will produce a call that satisfies the register assignment and still passes the wrong bytes, which is the subject of struct passing and register classification.

The narrow reading is the accurate one: extern "C" means “this function is named as C would name it and is callable under the platform’s C convention”. Everything else (who allocates, who frees, which thread may call, whether the function may unwind) is outside the promise and must be established separately. The symbol and linkage layer covers the naming half; the allocation half is covered by allocator mismatch.

Reading a call site as evidence

A disassembly of any call into a native library shows the contract directly. Consider a function that takes two integers and a double and returns a pointer:

#include <stdint.h>

/* Definition supplied by the native library; the caller sees only this. */
int64_t *lookup(int32_t key, int32_t flags, double weight);

int64_t *call_site(void) {
    return lookup(7, 0, 0.5);   /* edi=7, esi=0, xmm0=0.5, result in rax */
}

Compiled for System V AMD64, the call site loads edi with 7, esi with 0, xmm0 with 0.5, then executes call lookup. Nothing else is required: no shadow space, no stack adjustment beyond what alignment demands, no explicit save of the argument registers. The callee reads edi, esi, and xmm0, and writes its result to rax. The entire interface is four registers wide, and the compiler emits no code that describes it, because the convention is not data but a fixed agreement both sides were compiled against.

That is also why FFI mistakes are so rarely caught by type systems. The information needed to check a call across a boundary exists only in the platform ABI document, and the two sides are compiled separately, often by different toolchains, with no shared declaration to compare. The next layer down, designing a stable C ABI, is largely about making that invisible agreement explicit enough that it can survive a compiler upgrade on one side only.

Where the contract is thinnest

Three classes of call routinely violate the convention without violating any type check.

The first is the callback. A function pointer passed into foreign code is invoked by that code with whatever convention its own declaration assumed, on whatever thread the library chose. The receiving language must supply a trampoline that matches, and the details are covered in native callbacks into managed code.

The second is the variadic call. Argument promotion rules mean a float becomes a double and a small integer becomes int, so a language that passes its native types directly will hand over a value in the wrong width. There is no portable way to construct a variadic argument list at run time in C without the va_list machinery, and most FFI layers restrict variadic calls to a fixed set of signatures for this reason.

The third is the call that unwinds. The convention says nothing about what happens when a callee throws, because in C nothing throws. A language with exceptions that lets one propagate through a C frame relies on unwinding metadata the C compiler may not have emitted, and on a personality routine the C runtime does not have. That failure mode is treated in unwinding across an FFI boundary, and the rule it produces is blunt: every boundary either catches or aborts, and the choice must be made by the side that owns the stack frame.