Error Returns and the errno Convention

The most-used error-reporting mechanism in Unix is a global variable that is not really a variable, is not per-call, and does not exist in the kernel at all. errno is a thread-local integer that the C library sets after a failed system call, and the value it holds is a translated copy of a number the kernel placed in a register. Every layer of that description matters: the translation decides which values are visible, the thread-local storage decides what happens across a signal handler, and the fact that the kernel does not participate decides why some failures are indistinguishable.

The design is old enough to predate threads, and its retrofitting to threads is the source of most of the ways it can be used incorrectly.

A single register carrying two outcomes

The kernel has one register to report with. It must convey success or failure and, in either case, a value. The convention it uses is a reserved range: success returns a value that is either a small non-negative number, a count, or a pointer, and failure returns a negated error number in the range −1 to −4095. The kernel itself never sets errno; it has no such variable. It returns -EINVAL and lets the caller decide what to do.

The C library wrapper is what turns that into the familiar interface. Its shape is a few instructions:

/* Decoding a raw kernel return value: the check every libc wrapper
   contains, in the form the kernel contract requires. */
static long decode(long r)
{
    if ((unsigned long)r >= (unsigned long)-4095) {
        errno = (int)-r;      /* the register held a negated errno value */
        return -1;            /* the caller-visible failure */
    }
    return r;
}

The cast to unsigned long before the comparison is the idiom that makes one branch serve both cases. A valid descriptor is small and positive, so as an unsigned value it is far below the threshold. A pointer to a mapped object is also positive and below the kernel’s half of the address space, so it clears the threshold too. A negated error is a value just under zero, which as unsigned is just under 2^64, comfortably above. One unsigned compare replaces two signed comparisons, and there is no edge case at zero.

That −4095 boundary is a contract, not a coincidence. The kernel’s internal macro for the largest error magnitude is 4095, well above any error number it currently defines, so the band leaves room for new ones, and it bounds the range that success values must avoid. On x86-64 the band sits in the region of the address space reserved for the kernel, so a pointer cannot collide with it and mmap has the whole user range to work in. On an architecture where a user address could equal one of those values, an interface whose natural success values could land in the band would need a different reporting channel, a convention that shows up whenever a new call returns a pointer rather than an index.

What the wrapper costs, and why some calls skip it

The wrapper is not free in the sense that matters at the boundary. It adds a register move, a comparison, a conditional store to thread-local storage, and, on failure, a load of the thread pointer. On a call that costs hundreds of cycles that is noise. On the calls cheap enough for the wrapper to be a visible share of the total, the answer has generally been to remove the trap rather than to shrink the wrapper: getpid returns a value the kernel assigns once at process creation and never changes, so the C library can answer it from user space by caching it in thread-local storage without entering the kernel at all. The same optimisation cannot apply to getuid, whose value changes with setuid and its relatives, so the library has to ask the kernel each time. The reasoning that removes the trap extends further, to the code served from a page mapped into the process.

The vDSO is the main case: clock_gettime and gettimeofday are served by code mapped into the process, so there is no privilege transition at all and the wrapper described above never runs. That whole mechanism is treated in The vDSO and System Calls That Never Trap. Where a call still traps, the wrapper’s cost is dominated by the trap, and the error decoding is irrelevant.

There is a second reason to bypass the wrapper: it is the only place where the two-channel result is collapsed into one. A caller that needs the raw value, whether a tracer, a sandbox, or a runtime implementing its own threading, reads the register directly and interprets the range itself. The range is a matter of public contract for the Linux ABI, so doing so is well defined; doing so and then forgetting the range is the bug described below.

errno is per-thread, and that took work

The original interface had one global integer. With threads, two threads failing at once would race on it, so the standard requires that errno expand to a thread-local lvalue. On a typical implementation it is not a variable at all but a macro that calls an accessor returning the address of a slot in the thread control block. That is why errno cannot be assigned through a pointer captured before a fork or across a coroutine switch that the library does not know about, and why taking the address of errno in one thread and using it in another reads a slot belonging to neither.

Two consequences follow. The first is that errno is meaningless unless a call actually failed. A successful call is permitted to leave the previous value untouched, and many do, so code that reads errno after a success reads a stale value from an earlier failure. The portable discipline is to set errno to zero before a call that documents setting it, and to test the return value first. The second is that a signal handler interrupting a call may change the value, since the handler runs on the same thread and may itself make calls. Saving and restoring errno around a handler that might fail is standard practice for that reason.

/* A handler that preserves errno, because it runs on the same thread
   as the interrupted code and may perform its own failing calls. */
static void on_alarm(int sig)
{
    int saved = errno;
    (void)sig;
    /* ... work that may set errno ... */
    errno = saved;
}

Which failures the caller can and cannot tell apart

The range convention gives roughly four thousand distinct error numbers, and the kernel uses perhaps a few hundred. That sounds generous until the caller’s question is more specific than the number.

EAGAIN is the clearest case. It means the operation could not complete now and might complete later. It does not say which of the many conditions that produce it applies: whether the descriptor is non-blocking, whether a lock is held elsewhere, whether a resource limit was reached. EINTR means a signal arrived before the operation made progress, and the caller cannot learn which signal from the return value; the handler knows, the caller does not. EIO covers a wide band of device-level failures that the filesystem cannot classify further. ENOMEM may mean the process’s address space is exhausted, the system is out of pages, or a per-cgroup limit was hit; the number is the same in all three.

This compression is intentional. The kernel reports what it can establish cheaply and what the caller can act on. A richer channel would cost a copy out to user memory on every failure, and failure is not rare: a server polling thousands of descriptors sees EAGAIN constantly, and making it expensive would be worse than making it ambiguous.

The ambiguity is why some interfaces carry a second channel. waitpid can report a child’s exit status. read returning zero is end of file, distinct from any error. connect reports a failure asynchronously through the socket’s error state, which the caller retrieves separately. Where a distinction matters enough, the interface grows a way to express it rather than overloading an error number.

The paths where the convention is not the one described

Three deviations are worth knowing. A 64-bit program that issues int 0x80 reaches the 32-bit syscall table, not the 64-bit one, and receives its result through a 32-bit register. The error encoding survives that narrowing, because an error number fits comfortably inside 32 bits and the band means the same thing either way. What fails is the other direction: a success value wider than the register cannot survive being truncated, so a call that legitimately returns a 64-bit count, a 64-bit file offset, or a pointer above 4 GiB cannot be made through that door at all. This is the practical hazard of the mixed-width ABI described in 32-bit Compatibility System Calls on a 64-bit Kernel.

The second deviation is restart_syscall. When a call is interrupted by a signal after it has already performed part of its work, the kernel may be unable to simply return EINTR without losing progress, so it arranges for a subsequent call to resume. The C library hides this: a wrapper for a restartable call retries automatically unless the caller installed a handler without the restart flag. A program issuing raw syscalls must handle it, and a program that does not will see sporadic EINTR on calls the documentation describes as restartable. The interruption machinery, and the user-space critical sections that rely on a different part of the same kernel, belong to Restartable Sequences and Interruption Semantics.

The third is the kernel’s own internal convention, which is entirely different and entirely invisible from user space. Kernel functions return zero or a negative error number directly, with no errno, no wrapper, and a strong convention that a pointer-returning function encodes an error as a small negative value that must be checked with a dedicated macro before dereference. Confusing that with the user-space convention is a classic source of kernel bugs, and the discipline exists because the kernel has no thread-local storage to put an errno in, which is the very thing the C library provides on the other side of the boundary.

What the convention implies about interfaces

Because failure is a range and success is everything else, a new system call’s designers must decide what values it can legitimately return before they can decide it is well formed. A call that returns a count can never return a negative count. A call that returns an identifier must use values outside the error band. A call that returns a pointer must be defined so that the error band is unreachable, which is why kernel-allocated addresses and user-visible pointers live where they do.

That constraint propagates into the C library, which must decide whether a call is “restartable”, whether it retries, and what it returns to the caller. None of that is in the kernel’s source; it is in the wrapper, and it differs between libraries and between versions of the same library. The kernel’s contract is only the register and the range. Everything above it, the variable, its thread-locality, and the retry policy, is a convention layered on top, and each layer is a place where a caller can be wrong about what a failure meant.