io_uring and Shared-Memory Submission Queues

A server handling a hundred thousand operations per second through read and write spends a measurable fraction of its time not doing I/O. Each operation pays a privilege transition, copies the request and the result across the boundary, and is charged to the scheduler’s accounting. The work itself, moving bytes between a socket and a buffer, is often the smaller part. io_uring is the interface built on the observation that the transition and the copies are the cost, and that all three can be removed for the common case by putting the request and the completion in memory both sides can see.

The result is not a system call in the usual sense. It is a queue that user code writes to and a queue that the kernel writes to, with a small number of system calls used only when the process needs the kernel to look at the first queue or wait for the second.

Two rings and a shared memory region

The interface is built from three mappings over one region, established by a single call at setup time. The submission queue is an array of entries that user code fills in. The completion queue is an array of entries the kernel fills in. Around each array are two counters: a head and a tail, each an unsigned integer that only one side writes.

The counters are the whole synchronisation mechanism, and they are deliberately primitive. The kernel owns the head of the submission queue and the tail of the completion queue; user code owns the complementary halves. Because each counter has exactly one writer, no lock is needed and no atomic read-modify-write is required, only a store with release ordering on the producing side and a load with acquire ordering on the consuming side. That is the property that makes the ring cheap: the common path involves no atomic operation beyond the fences.

Entries are fixed size, which is what allows the array to be indexed by a counter masked to the ring size. A submission entry is 64 bytes; a completion entry is 16. The ring sizes are powers of two so that the mask is a bitwise and rather than a modulo. An entry being filled is therefore a single cache line on the common architectures, which matters because the kernel reads it across the boundary.

The counters wrap at their natural width, and because the ring length is a power of two, the distance between head and tail computed with unsigned subtraction gives the number of entries in flight even across a wrap. There is no separate count to maintain and no ambiguity when the ring is exactly full: the difference equals the ring size, which is the signal that no more entries can be added.

The call pattern, and where the traps went

A program that wants to submit operations writes entries into the submission array, advances the tail, and then, only if it needs the kernel to notice, issues one call telling the kernel that entries are available. It can also ask the same call to wait for completions. In the fully batched case the program submits thirty-two operations with one transition and reaps thirty-two completions with another.

/* The shape of a batched submit-and-wait, reduced to the ring arithmetic.
   Real code uses the library wrappers; the counters are the mechanism. */
struct ring {
    unsigned *sq_head, *sq_tail;
    unsigned *cq_head, *cq_tail;
    struct io_uring_sqe *sqes;   /* 64 bytes each */
    struct io_uring_cqe *cqes;   /* 16 bytes each */
    unsigned  mask;              /* ring size - 1 */
};

static void submit_one(struct ring *r, const struct io_uring_sqe *e)
{
    unsigned tail = __atomic_load_n(r->sq_tail, __ATOMIC_RELAXED);
    unsigned head = __atomic_load_n(r->sq_head, __ATOMIC_ACQUIRE);

    if (tail - head > r->mask)          /* ring full: must enter the kernel */
        return;

    r->sqes[tail & r->mask] = *e;
    __atomic_store_n(r->sq_tail, tail + 1, __ATOMIC_RELEASE);
}

static int reap(struct ring *r, struct io_uring_cqe *out)
{
    unsigned head = __atomic_load_n(r->cq_head, __ATOMIC_RELAXED);
    unsigned tail = __atomic_load_n(r->cq_tail, __ATOMIC_ACQUIRE);

    if (head == tail)
        return 0;                       /* nothing completed yet */

    *out = r->cqes[head & r->mask];
    __atomic_store_n(r->cq_head, head + 1, __ATOMIC_RELEASE);
    return 1;
}

The acquire and release orderings are load-bearing. Without the release on the tail store, the kernel could observe the advanced tail before the entry it points at was written, and would read a half-constructed request. Without the acquire on the head load, user code could read an entry the kernel has not finished writing. The counters are the only channel through which the two sides communicate, so every ordering rule in the interface is a rule about them.

The check for a full ring is where the design shows its hand. If user code fills the ring faster than the kernel drains it, the program must enter the kernel, not to submit but to let the kernel consume. That is a backpressure mechanism, and it is why a program that ignores the return value and keeps writing will silently drop requests or overwrite entries. The library wrappers handle this; a hand-written ring must not.

Kernel-side polling, and what it trades

The default mode still requires the program to issue a call to tell the kernel that entries are waiting. There are optional modes in which a kernel thread polls the submission ring instead, so the program submits without issuing that call at all: it writes entries and the kernel picks them up. The thread sleeps after a configurable idle interval and sets a flag in the ring asking to be woken, so the program still owes a call, but only when the ring has gone quiet. The trade is explicit and it is a good illustration of the cluster’s central theme.

In polling mode, the kernel spends CPU time looking at a ring that may be empty. That is a core consumed for the privilege of removing a transition per batch. It is a good trade when the ring is almost always non-empty, as in a busy server with hundreds of thousands of operations per second, and a bad one when the ring is idle most of the time, because the polling thread still runs. The mode also changes the failure semantics: a program that crashes while the kernel is polling leaves the kernel holding a reference to a ring whose process is gone, and the teardown path must handle that.

The polling thread does not remove the memory ordering requirements. It changes who observes the tail, not what the observer must do to see the entries safely.

The remaining copies

Removing the traps does not remove the data movement. A read operation still needs a buffer, and the kernel still needs to get the device’s data into it. Three cases exist.

If the buffer is a user address, the kernel must translate it and must keep the pages stable for the duration of the operation, which means pinning. Pinning costs a page-table walk per page and takes the pages out of the memory manager’s reach until the operation completes, so a program that submits a million small reads with distinct buffers pays a large amount of pinning work. Registering a buffer once and reusing it across many operations amortises that cost, which is why the interface provides a registration call: the pages are pinned once, and subsequent operations refer to them by index.

If the operation is a network receive and the hardware supports it, the data can be placed directly into a user buffer by the device, with no kernel copy at all. That requires the device to be able to write to the address, which requires the mapping to be stable, which is the same pinning problem in a different form.

And for filesystem operations the kernel may still copy metadata structures in and out, because the results are not a byte stream. The copy of a completion entry, sixteen bytes, is trivial; the copy of a directory entry or a status structure is not.

Ordering guarantees and their absence

A ring that permits many operations in flight forces the question of ordering. Two submissions may complete in any order, and the interface is explicit that completions carry an identifier chosen by the program so that results can be matched to requests. Without such a link, a program that submits two reads into the same buffer cannot tell which completion corresponds to which read.

Ordering between dependent operations is a separate problem. A read followed by a write of the data read cannot be expressed as two independent submissions, because nothing guarantees the second runs after the first. Implementations provide a way to chain operations so that one starts only after another completes, and the chain is expressed in the entry rather than enforced by the ring. The distinction matters: the ring provides no ordering, and any ordering a program needs must be built from the chaining facility or from waiting on completions before submitting dependents.

This is the same class of problem that the kernel faces internally with memory barriers, moved into an interface where the program is responsible for getting it right. A ring buffer between two parties with independent progress is a concurrency problem, and the counters alone do not solve it; they only make the handoff safe.

Failure modes

The interface’s failure modes are mostly teardown and accounting problems rather than I/O errors.

A program that exits while operations are in flight has its rings torn down by the kernel, and in-flight operations are cancelled. Completions that were already written may be lost, which is acceptable for a dying process but is a hazard for a library that closes a ring while another thread is still using it. The close path must be sequenced against the submitting threads, and the interface provides no automatic synchronisation for that.

A program that fills the completion ring and does not drain it stops receiving completions until it drains some. What happens in the interval depends on the kernel version: current kernels hold the pending completions internally and post them when space appears, while earlier ones refused further submissions with a busy error until the application had reaped from the ring. Neither outcome is an error at the call site that submitted the operation that could not be recorded, which is why the ring sizes must be chosen with the worst case in mind rather than the average.

Resource limits apply per process and per user, and exceeding them fails the setup call rather than individual operations. The limits cover the number of rings, the number of pinned pages, and the number of entries; a program that scales its ring count with its thread count will hit them.

And the interface is version-sensitive in a way that plain system calls are not. The set of operations grows with kernel releases, and an operation submitted to a kernel that does not implement it completes with an error rather than failing at submission. A program that assumes a specific operation exists must probe for it. The same applies to flags: an unknown flag is rejected at setup, which at least fails early, but the set of accepted flags differs across versions in ways that are not always documented per release.

Where this sits

The interface is the most complete answer to the cost described in What a System Call Actually Costs: it converts a per-operation transition into a per-batch transition, and it converts a per-operation copy into a registration cost amortised over many operations. It does not remove the boundary; it makes the boundary’s cost independent of the number of operations.

That is the same idea the vDSO applies to a narrower problem: removing the transition entirely where the answer needs no privilege, as described in The vDSO and System Calls That Never Trap. Between the two lies the whole design space: remove the transition, amortise it, or accept it. What determines which is available is not performance ambition but whether the operation needs the kernel to make a decision that only the kernel can make.