System Call ABIs and the Register Argument Convention

A function call and a system call look alike at the source level and share almost nothing below it. A function call passes arguments in registers or on the stack according to a convention both caller and callee were compiled against, and returns through a ret that the processor treats as an ordinary jump. A system call passes arguments in registers chosen by the kernel’s ABI, returns through a privileged instruction, and reports failure through a channel that has no counterpart in ordinary C. The conventions are close enough to be confusing and different enough that mixing them produces bugs which survive review.

The conventions are also per-architecture and, on some machines, per-personality. A single kernel binary can expose a 64-bit ABI, a 32-bit ABI, and an x32 ABI whose arguments sit in 64-bit registers but whose pointers and long are 32 bits wide. Each has its own entry point, its own numbering space and, in places, its own idea of what a structure looks like in memory.

Where the arguments go, and why only six

On x86-64 the System V ABI passes the first six integer arguments of a function call in RDI, RSI, RDX, RCX, R8 and R9. The kernel’s syscall ABI uses RDI, RSI, RDX, R10, R8 and R9. One register differs, and the reason is architectural rather than arbitrary: the syscall instruction overwrites RCX with the return address, so the fourth slot had to move. R10 is the nearest spare that is not otherwise clobbered by the instruction. A function that takes four integer arguments and is called both ways therefore needs a shuffle at the boundary, which is part of what a libc wrapper does.

Six is a convention, not a hard limit imposed by the hardware. It matches the number of integer arguments the function-calling ABI passes in registers, so a wrapper can move arguments across with minimal shuffling, and almost every interface the kernel needed fit inside it. On AArch64 the pattern is similar and tidier: the svc instruction does not clobber argument registers, so arguments arrive in X0 through X5, the syscall number in X8, and the result comes back in X0. On 32-bit x86 with int 0x80 the arguments go in EBX, ECX, EDX, ESI, EDI and EBP, a set chosen partly because it avoids registers the instruction itself touches.

The six-slot ceiling is visible in the interfaces. Calls that would naturally take more arguments are defined to take a pointer to a structure instead. mmap fits in six. Anything that grows, whether a structure describing a socket address or a batch of operations, is passed by pointer. This is not only about register count: a pointer argument lets an interface gain fields later without changing its signature, because the kernel can read a size or a flags word and decide how much of the structure to interpret.

Architecture	Number	Arguments	Result
x86-64	RAX	RDI, RSI, RDX, R10, R8, R9	RAX
x86, int 0x80	EAX	EBX, ECX, EDX, ESI, EDI, EBP	EAX
AArch64	X8	X0–X5	X0
RISC-V (64-bit)	A7	A0–A5	A0

The table is worth reading as a warning. The same source-level call compiles to four different instruction sequences, and a hand-written stub that assumes one will fail on the others.

Pointers, sizes and the width trap

An argument that is a pointer is a number whose interpretation depends on the ABI that produced it. A 64-bit process passes a 64-bit address; the kernel must validate that address against the calling process’s address space and must not be talked into treating a value above the user limit as a kernel pointer. A 32-bit process on the same kernel passes a 32-bit address that the kernel narrows from a register the hardware fills at 64 bits. The kernel therefore does not receive a pointer in any usable sense. It receives an integer, and the copy routines that turn it into a kernel object are aware of which ABI is in play.

Width mismatches go beyond pointers. A 32-bit ABI’s long is 32 bits while the 64-bit ABI’s is 64, and off_t follows the same split unless the interface opts into a wider type. A structure passed by pointer therefore has a different size and different field offsets in the two ABIs. The classic case is file status: the 64-bit structure describing a file’s metadata has larger inode and size fields and different padding than the 32-bit one, and the layout differs again between architectures because of alignment rules. Kernels handle this by defining one structure per ABI and converting on the way out, a translation layer that is mechanical, pervasive and easy to get subtly wrong: a field added to the 64-bit structure without a matching change to the 32-bit one leaks kernel stack bytes to a 32-bit caller.

The same problem appears without any 32-bit process being involved. Interfaces that were defined before 64-bit file offsets existed have a 32-bit variant and a *64 variant; the names and the existence of the split are visible in the man pages. Newer interfaces avoid the problem differently: they use explicitly sized types rather than long and off_t, and they put extensibility in a flags argument so that a structure’s shape need not change when behaviour does.

Return values and the two-channel result

A function returns one value. A system call returns two pieces of information: whether it succeeded, and either a result or a reason for failure. Since the return channel is a single register, the two must be encoded together.

The encoding is a range convention. Results that could plausibly be returned on success occupy the non-negative range and small negative values are impossible for them, so the kernel reserves a band of negative values for errors. On Linux that band is −1 through −4095: the kernel returns a negative errno value directly in RAX, and the C library wrapper checks the range, moves the value into errno and returns −1 to the caller. The band is wider than the set of error numbers the kernel currently defines, so new ones fit without changing the contract, and it is narrow enough that the test is a single unsigned comparison after negation: a value is an error if it is at or above the two’s-complement representation of −4095. That costs a comparison rather than a table lookup.

This design has a consequence worth stating plainly: a system call can never return a value in that band on success, so any interface whose natural success values could land there needs a different channel. On x86-64 the band sits at the top of the 64-bit address space, in the region reserved for the kernel, which user code cannot map, so an address-returning call has the whole user range available. On an architecture where a user address could equal one of those values, the constraint would bind and the interface would have to report success some other way. The wider question of how errors are reported, including the older int 0x80 behaviour where the error band differs, belongs to Error Returns and the errno Convention.

Numbering, stability and the dispatch table

The syscall number is an index, not a symbol. It is assigned once, recorded in a header that both the kernel and the C library include, and never reused even after the call is removed, because a binary compiled years ago still contains the old number and must receive a sane error rather than whatever the slot has become. Numbers are per-architecture: x86-64 and 32-bit x86 have different tables and the same conceptual operation has different numbers in each, which is why a syscall number cannot be hardcoded portably.

Dispatch is a bounds check and an indirect call. The kernel compares the number against the table length, and if it fits, calls the function pointer stored at that index. The check must come first; without it, a number chosen by user code becomes an out-of-bounds read of kernel memory followed by a jump to whatever it contains. The table itself is generated from a list at build time so that the numbering and the function set cannot drift apart, and the generation step also produces the stubs that marshal arguments, which is where the register shuffling described above actually lives.

New calls are added under a rule that is worth knowing because it explains the shape of recent interfaces: a new call must be usable by a process that does not know it exists, must not need a privileged capability to be useful in the common case, and must be extensible through a flags argument rather than by adding parameters. The flags word is the escape hatch that keeps the six-argument ceiling from being a real constraint.

When the ABI is not the one the caller assumes

Three situations produce a mismatch. A statically linked binary built against old headers passes structures the running kernel has since extended; nothing repairs that in general, but calls whose semantics were extended through flags rather than through structure growth still work, and calls that were later given a size argument or a new variant have a way to say what the binary understands. A binary for a different ABI, such as 32-bit on a 64-bit kernel, enters through a different instruction and is dispatched to a translation wrapper, not to the native handler. And a call issued through a language runtime’s own convention, such as a syscall instruction emitted directly rather than through the C library, bypasses the wrapper’s argument shuffling and error decoding, so the caller must reproduce both.

The last case is where most hand-written syscall code fails: it gets the register assignment right and then treats a negative return as a value. The kernel’s contract is precise: the value in the result register, negated, is the error number. Nothing in the hardware enforces it. What the hardware does enforce, and what every one of these conventions rests on, is the privilege transition itself, described in How a System Call Transfers Control to the Kernel.