Where a read() Syscall Spends Time Before It Reaches the Data

A read of one byte and a read of four kilobytes execute nearly the same kernel code. The difference between them is a memory copy whose size is the only thing that varies. Everything else — the privilege transition, the descriptor lookup, the permission checks, the file operation dispatch, the page cache probe, the return path through pending-signal checks — happens regardless of how many bytes were asked for. That is why a program that issues one syscall per byte can spend most of its life in the kernel and still move almost no data, and why the first thing to establish about an I/O path is how many calls it makes per unit of payload.

The entry and exit sequence is unconditional

On x86-64 a system call begins with the syscall instruction, which loads the kernel entry address from a model-specific register and switches to ring 0. The kernel entry stub saves the user register state into a pt_regs structure on the kernel stack, swaps the per-CPU kernel data pointer, and, when page table isolation is active, switches to the kernel page tables. The number in the designated register selects a handler from the syscall table.

None of that work is charged to the payload. Neither is the return path, which restores the saved state, checks the thread’s flag word for pending work such as a signal delivery or a rescheduling request, and returns with sysretq. The exit check matters for latency because it means a syscall is a preemption point even when it did nothing interesting.

Hardening features sit directly on this path. Page table isolation forces a CR3 reload on each transition. Speculation mitigations add barriers and thunked indirect calls. A seccomp filter, if installed, runs a BPF program against the syscall number and arguments before dispatch. Each mechanism is individually modest; together they turn the cheapest possible syscall into a cost measured in hundreds of nanoseconds to a few microseconds depending on the processor and configuration. The exact figure is machine-specific, but its order of magnitude is what makes small I/O expensive: at a few hundred nanoseconds of fixed cost, a syscall that transfers 64 bytes achieves a ceiling far below the memory bandwidth of the machine.

Descriptor to file object

The first argument is an index into the calling process’s descriptor table, held in the files_struct attached to the task. The lookup is a bounds check against the table’s current size followed by an array read, protected by RCU so that the common case needs no lock. An index at or beyond the allocated size returns EBADF, as does a slot holding no file.

The result is a struct file, which carries the file position, the open flags, a reference to the inode and dentry, and a pointer to the file_operations table that implements the object’s behaviour. Two consequences follow. First, the descriptor table is per-process, so threads sharing a descriptor table contend on the same entries only when they mutate it; reads of an existing entry do not serialise. Second, the operation table is polymorphic: the same read call reaches a regular file, a pipe, a socket, a terminal or a device through different functions, and the cost of the call depends on which one is installed.

Between the lookup and the handler, the kernel validates the buffer range and the count. The count is clamped to a maximum transfer per call, which on Linux is just under two gibibytes (MAX_RW_COUNT, the largest page-aligned value below INT_MAX); a larger request is not an error, it simply transfers less than asked. Access control hooks run next: the file mode is checked against the requested operation, and any stacked security module gets a chance to deny it.

Two copies and a cache probe

For a regular file the handler is the generic file read path. It consults the page cache, an index from file offset to physical page, before touching the device. On a hit the kernel copies from the cached page into the user buffer with a routine that is aware that the destination may be unmapped. On a miss it allocates a page, submits a read request to the block layer, waits for completion, and then performs the same copy.

The copy is the only part proportional to the request size. It runs at memory speed, on the order of tens of gigabytes per second for large contiguous buffers, with a per-call setup cost for very small ones. The cache probe is a radix-tree or xarray lookup whose cost grows logarithmically with file size, effectively constant for realistic files.

The copy also has a failure mode that shapes correct code. If the user buffer is partly unmapped or protected, the copy routine does not fault the process; the fault is caught by an exception table entry, the copy stops at the boundary, and the syscall returns the number of bytes transferred so far. A return value smaller than the request therefore means end of file, a signal interruption, or a buffer that could not be fully written. Distinguishing these requires inspecting errno and the offset, which is why a single unchecked read call is not a correct way to fill a buffer.

#include <errno.h>
#include <unistd.h>

/* Read exactly len bytes, or as many as exist before end of file. */
ssize_t read_full(int fd, void *buf, size_t len)
{
    size_t done = 0;
    char *p = buf;

    while (done < len) {
        ssize_t n = read(fd, p + done, len - done);
        if (n < 0) {
            if (errno == EINTR)
                continue;          /* interrupted before any byte arrived */
            return -1;
        }
        if (n == 0)
            break;                 /* end of file */
        done += (size_t)n;
    }
    return (ssize_t)done;
}

Why the cost curve is steep at the left

Plot cost per byte against request size and the curve falls sharply and then flattens. At 16 bytes the fixed cost dominates by two or three orders of magnitude. At 4 KiB the copy is comparable to a fraction of the fixed cost. At 1 MiB, with readahead and page-cache hits, the per-byte cost approaches the memory copy cost and the syscall overhead becomes a rounding error.

The practical implication is arithmetic rather than intuition: moving one gibibyte in 4 KiB requests requires 262,144 syscalls; in 1 MiB requests it requires 1,024. At an approximate microsecond of fixed cost per call, the difference between those two strategies is measured in hundreds of milliseconds of pure kernel entry work, independent of the device. The device may be the larger term, but the syscall count is the term that can be removed without changing hardware.

Interfaces that amortise the transition

Three families of interface attack the fixed cost.

The first keeps the syscall count constant per buffer while changing what happens inside. readv and writev transfer up to IOV_MAX segments in one call, which on Linux is 1024; preadv and pwritev add an explicit offset so no separate seek is needed. The kernel still copies each segment, but it resolves the descriptor and the file once.

The second removes the descriptor lookup entirely. With io_uring, a submission queue entry carries an index into a table of pre-registered files and buffers, so a request does not need to translate a descriptor number, take a reference, or pin a user page. The batch of submissions is handed over with one call, or with none at all when a kernel polling thread is configured. The ring protocol is the subject of io_uring Submission and Completion Ring Mechanics.

The third removes the syscall from the data path. Memory-mapped files replace read calls with page faults, so the transition cost is paid once per page touched rather than once per request, and repeated access to a resident page costs nothing at all. The trade is that faults are harder to batch and that truncation of the mapped file turns subsequent access into SIGBUS. Zero-Copy Paths: What sendfile, splice, and mmap Remove covers where the copies go.

Where the fixed-cost model stops predicting

The model assumes each call is independent. That fails under concurrency, where the kernel work of one task competes with others for the same cores and the same caches, and the effective cost per syscall rises with the number of threads issuing them. It also fails when the call blocks: the measured latency of a read that misses the page cache is dominated by the device, and the syscall overhead is invisible inside that wait.

A third exception is the descriptor-table mutation path. Installing or closing descriptors takes locks and, in the shared case, invalidates per-thread caches of the table; a workload that opens and closes a file per request pays a cost several times the transition itself. The remaining chapters of this path — the write side, the page cache policy that decides whether a read hits, and the durability calls that force data out — each build on the fixed cost described here rather than replacing it.