Readiness and Completion: Two Ways to Ask the Kernel for I/O

Two questions can be asked about a socket or a file. The first is whether an operation on it can proceed without blocking. The second is whether an operation that was started has finished. The first is readiness; the second is completion. Both are answered by the kernel, both are used by production runtimes, and the difference between them determines the shape of the runtime that consumes them.

An event loop built on epoll asks the first question. The kernel says a descriptor is readable; the program calls read, which copies bytes into a buffer the program chose, and the program handles a short count if the transfer was partial. An event loop built on io_uring asks the second. The program submits a request naming a buffer and a length, the kernel performs the copy, and a completion entry reports how many bytes moved and whether the operation succeeded.

Readiness costs a syscall per operation

In the readiness model, the notification is separate from the transfer, and both are system calls. A connection that receives a request and sends a reply performs at least one epoll_wait return, one read, one write, and, if the write is partial, another write and an epoll_ctl to arm the write side. The handler logic is straightforward, but the syscall count is two per operation at minimum, and each syscall costs on the order of a microsecond once Spectre-era mitigations are in place.

The model also requires the program to hold the buffers. Because read copies into a buffer the program supplies, the program must decide how much to read before knowing how much is available, and it must keep that buffer alive across the handler’s lifetime. For a server holding ten thousand connections, that is ten thousand receive buffers, each at least large enough for the largest expected message, unless the runtime uses a fixed pool with a copy step.

The compensating advantage is that the program keeps control. Partial transfers are visible as they happen; a handler can process the first kilobyte of a message while the rest is still arriving. There is no shared memory protocol between program and kernel, no ring to size, and no registration step. The whole interface is three syscalls, and the loop that consumes it is the one described in readiness notification and the single-threaded event loop.

Completion inverts who owns the buffer

In the completion model the kernel owns the buffer for the duration of the operation. The program submits a request that names an address and a length; until the completion arrives, it must not write to or free that memory. This is a real obligation, not a formality: the kernel may be writing into the buffer at any moment, and a program that reuses it commits a data race that no lock protects against.

Two consequences follow. The first is that buffer lifetime becomes a first-class runtime concern. The runtime must either allocate a buffer per in-flight operation and reclaim it on completion, or use a registered buffer pool whose pages the kernel has pinned, in which case the runtime must track which buffers are in flight and which are free. io_uring supports registering a buffer set so that requests can refer to buffers by index instead of address, which saves the kernel from looking up and pinning pages per request.

The second is that the operation’s parameters are fixed at submission. A read request names a length before the data arrives, so the kernel fills up to that length and reports the actual count. A program cannot adjust the read in response to the first bytes it sees, because the first bytes are not visible until the whole request completes. This is the fundamental expressive difference: readiness lets a program react to partial progress, completion makes progress atomic.

The classic example is a protocol with a length prefix. Under readiness, a program reads the four-byte prefix, learns the body length, allocates exactly that, and reads the body. Under completion, a program either submits a fixed-size read and then a second request for the remainder, or reads into a buffer large enough for the maximum message and copies out the relevant part. The first approach serialises two round trips through the completion queue; the second wastes memory per in-flight request.

Rings, entries, and the shared-memory protocol

io_uring implements completion with two rings in a shared memory mapping: a submission queue and a completion queue, each with an array of entries and a pair of indices. The kernel and the program agree on the layout and communicate by updating head and tail counters with the appropriate memory ordering, so that in the steady state the program can submit work without a syscall at all. The kernel polls the submission queue when it has been told to, or the program calls io_uring_enter to submit and optionally wait.

struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fd, buf, len, offset);
sqe->user_data = tag;             /* copied into the completion entry */
io_uring_submit(&ring);           /* may or may not enter the kernel */

/* Later: reap completions in submission order of completion */
struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe);
handle(cqe->user_data, cqe->res); /* res is bytes moved or -errno */
io_uring_cqe_seen(&ring, cqe);

The user_data field is the runtime’s hook: it is an opaque 64-bit value the kernel copies back untouched, so a runtime stores a pointer or an index there and recovers its own per-operation state on completion without a lookup keyed by descriptor. That is a structural improvement over epoll, where the data field is per-descriptor rather than per-operation, so two operations on the same descriptor cannot be distinguished.

The ring depth is a fixed resource. Each entry in flight consumes a slot in the submission queue and a corresponding slot in the completion queue, and the kernel may refuse submissions when the queue is full. A runtime sizing a ring must decide how many operations it intends to have outstanding, which is the same decision as choosing a thread pool size but with different consequences: too shallow and the runtime stalls waiting for a free slot; too deep and the memory for entries and buffers grows without improving throughput.

What each model does to the scheduler

Readiness fits naturally into the task-per-connection design. A task suspends when it calls a read that cannot proceed, and the runtime registers interest in the descriptor. When the descriptor becomes ready, the task is made runnable. The scheduler never sees an operation, only a task; the kernel’s job is to tell the runtime which task can proceed.

Completion changes that. The unit of suspension is the operation, not the task, and one task may have several operations outstanding. The runtime must keep a table from completion tag to the task waiting on it, and a task that issues three reads concurrently has three entries. When all three complete, the task resumes. This makes the runtime’s own data structures the bookkeeping layer, and it makes the model a natural fit for a scheduler that wants to batch: submit a hundred operations, wait once, dispatch a hundred completions.

Completion also changes the cost of a syscall boundary. The ring is shared memory, so submission and reaping can both be done without entering the kernel when the kernel is already polling. The program’s cost per operation drops to a few memory writes with fences, and the kernel’s cost drops to reading the queue. The limit is that the kernel must be actively polling, which consumes a core; runtimes choose between polling mode, where a core spins on the rings, and interrupt mode, where a core sleeps until an entry arrives. The first gives lower latency at the cost of a busy core; the second is the same wakeup economics as epoll.

Portability and the absence of a common interface

The two models do not map onto each other cleanly, and no portable interface provides both. epoll and kqueue are readiness. io_uring is completion, on Linux only. Windows has overlapped I/O with completion ports, which is completion and has been available far longer. POSIX asynchronous I/O as specified is a third shape, with a notification mechanism that may be a signal or a thread callback, and its Linux implementation has historically been limited to a subset of operations.

A runtime that targets several platforms must therefore choose an abstraction that is expressible in both. The usual choice is readiness, because a completion can be simulated by performing the operation non-blockingly and reporting the result, whereas readiness cannot be simulated by a completion interface without the runtime issuing the transfer itself. The cost of that choice is that Linux-specific runtimes which use io_uring do better on high-throughput storage workloads, and portable runtimes do not.

The other direction is possible. A completion interface can be built on readiness by having a dedicated thread perform blocking operations and post results, which is exactly what a thread pool behind a readiness loop does. That design gives up the syscall reduction and keeps the programming model. The runtime consequence is a queue of pending operations and a set of threads, and the interaction between those threads and the scheduler is where blocking calls inside an asynchronous runtime becomes a correctness question rather than a performance one.

Which to build on

Readiness is the right base when the workload is mostly sockets, when the runtime must be portable, and when handlers benefit from seeing partial data. Completion is the right base when the workload is mostly storage, when the runtime can afford platform-specific code paths, and when operations are large enough that the per-operation overhead of a ring entry is small relative to the transfer. The two can coexist in one runtime, with a per-descriptor or per-operation choice, at the cost of two code paths and two buffer disciplines.

Neither model removes the need for the scheduler to know when a task can run. The notification tells the kernel’s story; the runtime’s job is to translate it into a scheduling decision, and that translation is the same problem whether the news is readiness or completion.