How a System Call Transfers Control to the Kernel

A program that calls write does not call a function in the ordinary sense. The code that will run belongs to a different privilege level, sits at an address the caller cannot execute, and expects to find the machine in a state the caller is not allowed to construct. No amount of linking or dynamic symbol resolution changes that. The transfer has to be an event the processor itself recognises, because only the processor can change the current privilege level; user code cannot assign itself a higher one any more than it can turn off paging.

On x86-64 that event is the syscall instruction, and its entire behaviour is defined by four model-specific registers that the kernel writes once during boot. Understanding those registers explains most of the oddities of kernel entry code: why a stub does almost nothing before switching stacks, why there are two return paths rather than one, and why the entry sequence is one of the most security-sensitive stretches of code in the system.

Protection levels and what the hardware enforces

x86 defines four privilege levels, and the current level is a two-bit field in the CS selector called CPL. Nearly every general-purpose operating system uses two of them: level 3 for user code, level 0 for the kernel. The distinction is enforced by the memory-management unit and by instruction decoding, not by convention. A load from a page whose page-table entry does not permit user access faults when CPL is 3, regardless of what the code believes. An attempt to write a control register, execute an I/O instruction, or load the page-table root faults when CPL is 3. hlt faults. lgdt faults.

That leaves exactly one question: how does execution legitimately arrive at level 0? The architecture provides two doors. One is an interrupt or exception, which is asynchronous from the program’s point of view and is delivered through the interrupt descriptor table. The other is a deliberate call gate, which on modern systems means the syscall instruction or, in older code, the int instruction with a software vector. Both doors are visible to user code, so both must be safe against a caller that supplies hostile register values. This is the central design constraint of every entry path.

The four registers that define the entry

The syscall instruction reads no memory and consults no descriptor table. It consults MSRs.

IA32_STAR holds two selector fields. The upper half supplies the CS and SS selectors that sysret will load when returning to user mode; the lower half supplies the CS selector loaded on entry, with SS implied as the next selector. IA32_LSTAR holds the single 64-bit address that RIP receives. That is the whole entry point: one address, fixed at boot, with no table to index. IA32_FMASK holds a mask of RFLAGS bits to clear on entry; the kernel uses it to clear the interrupt flag, the direction flag, the trap flag and the alignment-check flag, so that the first instruction of the entry stub runs with interrupts disabled and with string instructions pointing the safe way. IA32_KERNEL_GS_BASE holds the kernel’s per-CPU base, exchanged with the user’s GS base by the swapgs instruction.

When user code executes syscall, the processor performs a fixed sequence. RCX receives the address of the instruction after the syscall, which is the return address. R11 receives the current RFLAGS. RIP receives IA32_LSTAR. RFLAGS is masked with IA32_FMASK. CS and SS are loaded from the STAR field and CPL becomes 0. Two general-purpose registers are clobbered by the architecture itself, which is why the syscall ABI reserves RCX and R11 and passes the fourth argument in R10 instead of RCX.

The sequence conspicuously does not include a stack switch. RSP still holds the user’s stack pointer, and the kernel’s first instruction is executing at level 0 with a pointer it does not control and must not trust.

Why the first instruction runs on the user stack

Because interrupts are masked by FMASK, the window in which the kernel is running on a user stack is not interruptible by ordinary device interrupts. It is interruptible by non-maskable interrupts and machine checks, which ignore the interrupt flag. That asymmetry dictates the shape of the stub.

The stub’s first action is swapgs, which exchanges the GS base with the value in IA32_KERNEL_GS_BASE, giving the kernel access to its per-CPU data. Its second action loads a kernel stack pointer from that per-CPU data, discarding the user’s value. Only then can it push anything. Between those two instructions the machine is in a mixed state: privileged, on a user stack, but with kernel GS. An NMI arriving in that window finds a handler that must decide, without being told, which of the two states it interrupted. Kernel implementations solve this by saving both GS bases and comparing them against the per-CPU value, and by keeping the NMI entry path separate from the syscall path so that it never assumes the swap has happened.

The reverse window is just as delicate. If an NMI arrives before swapgs executes, GS still points at user-controlled data, so the handler must not touch per-CPU state through GS until it has established that it is on the user GS base. A handler that gets this wrong lets user code choose where the kernel reads and writes. Attacks against exactly this window have been demonstrated repeatedly, and the mitigations are why entry code contains several near-duplicate stubs rather than one shared prologue.

Nothing in the stub dereferences the user’s stack pointer. It records it, because the saved frame must describe the interrupted context faithfully for signal delivery and for ptrace, but it never uses it as a stack. That distinction matters: the value is attacker-controlled, and a kernel that pushed onto it would be writing to an address chosen by the caller.

Building the frame the rest of the kernel expects

After the stack switch, the stub pushes registers in a fixed order to build struct pt_regs. On x86-64 that structure holds twenty-one eight-byte slots: the fifteen general-purpose registers plus orig_ax, which records the original syscall number, and the five-entry machine frame of instruction pointer, code segment, flags, stack pointer and stack segment. The syscall entry path does not load those last five from the hardware, because the instruction leaves them where they were; it pushes the values it needs to describe the interrupted context. The layout is not a private implementation detail. Signal frames, core dumps, ptrace and the unwinder all consume it, so the order is effectively part of the kernel’s internal ABI and cannot be rearranged for convenience.

/* Issuing a system call with no libc wrapper: the raw instruction,
   showing which registers the hardware itself destroys. */
static long raw_getpid(void)
{
    long ret;
    __asm__ volatile ("syscall"
                      : "=a"(ret)          /* result arrives in RAX */
                      : "a"(39L)           /* __NR_getpid on x86-64 */
                      : "rcx", "r11", "memory");
    return ret;
}

The clobber list is not defensive habit. RCX and R11 are overwritten by the instruction before the kernel runs a single line of C, so a compiler that assumed they survived would corrupt the caller’s state. The memory clobber tells the compiler that the call may have observed or changed memory, which is true of every system call in principle.

From pt_regs, the C handler reads the syscall number from the saved RAX, bounds-checks it against the size of the dispatch table, and calls the corresponding function with arguments taken from RDI, RSI, RDX, R10, R8 and R9. Everything the hardware did not do is now the handler’s responsibility: validating pointers, copying structures in from user memory, checking that a file descriptor refers to something that supports the requested operation. The processor guaranteed only that CPL changed and that RIP is inside the kernel.

Two exits, and why the fast one is conditional

Returning is not the mirror image of entering, because the fast return instruction has preconditions the kernel cannot always satisfy.

sysret loads RIP from RCX and RFLAGS from R11, restores the user selectors from STAR, and drops to CPL 3. It is a handful of cycles. But it requires that RCX hold a canonical address. If it does not, the fault is raised at CPL 0, on the kernel’s stack, which is a much worse position than a fault in user code. sysret also forces certain flag bits rather than restoring them verbatim, and it cannot be combined with the page-table switch that kernel page-table isolation requires on parts vulnerable to speculative reads of kernel mappings. And before any return, the kernel must check whether pending work exists: a signal to deliver, a task to reschedule, a single-step trap to take. Each of those needs the full frame.

So the exit path branches. If the fast conditions hold and no work flags are set, sysret runs. If any of them fails, the kernel takes the general return path, which reloads CR3 when isolation is active and executes iretq, popping the five-entry machine frame and performing a full privilege return. The two paths must produce identical observable state, and keeping them in agreement is a recurring source of bugs precisely because the fast path is exercised constantly and the slow path almost never is.

The older int 0x80 door is still present on 64-bit kernels for 32-bit binaries. It is slower for structural reasons: it reads a gate in the interrupt descriptor table, switches stacks through the task-state segment, and saves a larger frame, and it cannot deliver a 64-bit return address. It survives because the 32-bit ABI is defined in terms of it, a subject taken up in 32-bit Compatibility System Calls on a 64-bit Kernel.

Where the transition breaks

Three failure classes recur. The first is the GS window described above, where an asynchronous event arrives at the wrong point in the stub. The second is frame desynchronisation, where the fast and slow return paths disagree about a saved register, producing corruption that appears only when a signal arrives during a specific syscall. The third is the assumption that the syscall number is valid: an out-of-range number must produce an error, not a jump through a wild pointer, which is why the bounds check precedes the table read and why the table is a fixed array rather than a computed offset.

The transition itself is cheap. The syscall and matching sysret instructions are a fixed, short hardware sequence, and the cost they add is small next to what follows. Everything that makes a system call expensive happens after the stub has built its frame and before the return path is chosen, which is the subject of What a System Call Actually Costs. The registers that carry arguments across the boundary, and the conventions that decide which ones, are covered in System Call ABIs and the Register Argument Convention.