32-bit Compatibility System Calls on a 64-bit Kernel

A 64-bit kernel that can execute 32-bit binaries is running two operating systems’ worth of interface on one set of page tables. The programs on one side have 32-bit pointers, 32-bit long, and a structure layout that predates 64-bit machines by decades. The programs on the other side have none of those. Both reach the same kernel, through the same privilege transition, and the kernel must know which kind of caller it is serving before it touches a single argument.

The arrangement is more than a set of wrapper functions. It is a parallel entry path, a parallel numbering space, a parallel set of data structures, and a translation layer that converts between the two layouts on every call that passes a structure. Where the layer is incomplete, the result is either a compatibility break or an information leak, and the history of the interface contains both.

Two entry paths through one instruction

On x86-64 the native path uses syscall, which delivers a 64-bit return address in RCX and a 64-bit flags word in R11. A 32-bit program cannot use it: its addresses are 32 bits, its code lives below the 4 GiB boundary, and a value above that boundary in RCX would be truncated the moment the program used it as a pointer. So the compatibility path uses int 0x80 instead, the older software-interrupt entry that a 32-bit kernel would have provided natively. A kernel can also choose to serve 32-bit calls through the syscall instruction, translating the register layout on the way in, but int 0x80 is the entry a compatibility binary contains.

The two paths differ below the instruction as well. int 0x80 reads the interrupt descriptor table, switches stacks through the task-state segment, and builds a frame in 32-bit layout. The kernel’s entry code branches on which instruction arrived, and the branch determines which handler table the syscall number indexes and which pt_regs layout the rest of the kernel will read.

The consequence is that the syscall number is not enough to identify an operation. The 64-bit and 32-bit tables assign numbers independently, so the same number selects a different call in each, and a binary’s idea of “syscall 1” differs from the other ABI’s. The kernel cannot dispatch on the number alone; it must also know the personality of the caller, and that personality is established by the entry path and recorded in the task structure. Every handler that takes a structure pointer, and every handler that returns one, consults it.

A third personality exists on x86-64: the x32 ABI, which uses the 64-bit syscall instruction and 64-bit registers but defines pointers and long as 32 bits. Its syscall numbers are the native ones with a distinct high bit set, so the number itself carries the personality. The design keeps the numbering spaces from colliding, and it means a dispatch that ignores that bit cannot tell the two 64-bit-register personalities apart.

What the translation layer does

For a call that takes no arguments beyond integers, translation is a no-op and the same handler serves both personalities. Most calls are not like that.

A pointer argument arrives as a 32-bit value in a register whose upper half may contain anything. Before the kernel dereferences it, the value must be narrowed to 32 bits, and the result must be checked against the caller’s address limit. For a 32-bit process that limit is 4 GiB, not the 64-bit kernel’s much larger user limit. The check is not optional: a 32-bit process that passes 0xFFFFFFFF as a pointer is asking the kernel to read from an address that is invalid in its address space, and the kernel must return an error rather than treat the value as a valid 64-bit address.

For a call that takes a structure, the layer must convert the layout. A 32-bit structure and its 64-bit counterpart have different field sizes and different padding, so the conversion is field by field, not a copy. The classic example is file status: the structure a 64-bit caller receives holds a 64-bit inode number, a 64-bit size, and 64-bit timestamps, while the one a 32-bit caller receives holds narrower fields laid out for the compatibility ABI. A call that returns file status therefore cannot return the 64-bit structure to a 32-bit caller; it must build the 32-bit structure and copy that out.

/* The shape of a compat wrapper: translate in, call the native
   implementation, translate out. The two directions are separate
   because a structure may be input-only, output-only, or both. */
static long compat_do_stat(const char __user *path, void __user *ubuf)
{
    struct kstat64 kst;
    struct compat_stat cst;
    long err;

    err = vfs_stat(path, &kst);        /* native implementation */
    if (err)
        return err;

    memset(&cst, 0, sizeof(cst));
    cst.st_dev    = kst.dev;
    cst.st_ino    = kst.ino;           /* may truncate: see below */
    cst.st_mode   = kst.mode;
    cst.st_size   = kst.size;          /* 32-bit field */
    /* ... remaining fields, each with its own width ... */

    if (copy_to_user(ubuf, &cst, sizeof(cst)))
        return -EFAULT;
    return 0;
}

Every one of those field assignments is a place where a value can be lost. An inode number larger than 32 bits truncates. A file size larger than 4 GiB truncates, which is why the interface acquired wider variants that carry 64-bit fields, and why a program compiled against the old headers silently sees wrong sizes for large files unless it was built with the large-file extensions that switch the interface to the wide version.

The leak that padding creates

The structure-building pattern above is where the compatibility layer’s security history lives.

A 32-bit structure is typically smaller than its 64-bit counterpart and has its own padding rules. If the kernel builds the 32-bit structure on its stack and does not initialise every byte, the padding bytes contain whatever was on the kernel stack. Copying the structure out to user memory then hands those bytes to the caller. The information is not privileged in any deep sense; it is stale kernel stack, which may contain pointers, keys, or fragments of another process’s data that passed through the same stack.

This is a recurring bug class, and its recurrence is structural: every new structure added to the 64-bit interface needs a 32-bit counterpart, and every counterpart needs a matching initialisation. The memset in the fragment above exists for this reason, not for tidiness. Kernels also handle it by converting field by field into a structure that was zeroed at declaration, and by refusing to copy out any structure whose layout they have not explicitly constructed.

The complementary hazard is a structure that grows. If the 64-bit structure gains a field, the 32-bit counterpart must gain it too, and a caller that passes a smaller buffer than the call writes will have the kernel write past the end of the caller’s buffer. Interfaces avoid this by taking a size argument the caller fills in, by defining success to depend on a version or flags field the caller sets, or by having separate calls for the two personalities. An interface that does none of the three cannot be extended safely, and adding a field to it is a compatibility break.

Where the compatibility path costs more

The path is slower than the native one, and the reasons are worth separating because only some of them are removable.

The entry instruction is the first. int 0x80 is structurally more expensive than syscall: it reads the descriptor table, performs a stack switch through the task-state segment, and saves a larger frame. That difference is fixed by the architecture and cannot be optimised away.

The dispatch is the second. A compatibility call goes through a table of wrappers, each of which may translate arguments, call a native implementation, and translate results. The wrapper’s work is proportional to the number of structure fields, so a stat costs more than a getpid.

The third is the most interesting because it is not inherent: the 32-bit address space is small enough that some operations are constrained by it. Mapping a large file into 32-bit address space requires either splitting the mapping or accepting that it cannot be mapped whole, and the code that handles the split is more complex than the 64-bit case. A structure that contains a pointer and is passed to a 32-bit caller must be built in the narrow layout even when the underlying object is larger, which means the conversion can involve truncation checks that the native path does not need.

None of these costs appear in a program’s source. They appear as a difference between two binaries built from the same source, and the difference is structural rather than marginal: a compatibility binary performs more work per call on calls that pass structures, and the same work on calls that pass integers.

Deciding which ABI a call belongs to

The kernel’s dispatch must make this decision before it reads the arguments, and the decision comes from state recorded at entry rather than from anything in the call itself. The entry stub determines which personality arrived from the instruction and the mode, and the handlers that care read that state; handlers that do not are shared between personalities.

The state matters for more than dispatch. It determines the address limit used by the copy routines, so a compat handler that checks against the wrong limit can be tricked into accepting an address that a 32-bit caller is not allowed to reach, a bug with a long history, because the limit check and the dispatch are separate steps and only one of them is obviously security-relevant.

It also determines what the signal frame looks like. A 32-bit process receiving a signal must receive a frame in 32-bit layout, and the kernel’s signal delivery code branches on the same flag. A mismatch between the dispatch personality and the signal personality produces a process that crashes on a signal it should have handled.

What the arrangement buys

Running 32-bit binaries at all is the obvious answer, and it is worth remembering why it is worth the complexity: an installed base of software compiled before 64-bit machines existed, and a class of programs for which 32-bit pointers are a space saving rather than a limitation. The compatibility layer is what lets one kernel serve both without a second kernel or a virtual machine.

The cost is a permanent parallel interface. Every structure has two layouts, every call has up to three numbering spaces, every handler that touches a pointer has a width to check, and every new feature must decide whether it applies to the compatibility personality at all. The layer cannot be removed while the binaries exist, and the binaries persist because running them on a 64-bit kernel required no change to those binaries when the kernel was written.

For a caller, the practical consequences are narrower than the mechanism. Results arrive in a narrower register, as described in Error Returns and the errno Convention. The register assignment differs, as described in System Call ABIs and the Register Argument Convention. And the numbering differs, which is why a syscall number is never portable across personalities. What does not differ is the privilege transition itself, the mechanism in How a System Call Transfers Control to the Kernel is the same event, entered through a different door.