io_uring Submission and Completion Ring Mechanics

Ordinary asynchronous I/O on Linux was expensive because the kernel never learned what the application already knew. Each operation was submitted with a syscall, each described by a structure the kernel had to copy in, and each buffer had to be translated from a virtual address to pages the device could use. io_uring replaced that with two rings of shared memory and a small set of operations that register resources once. The request that changes the arithmetic is that a batch of operations can be submitted without any syscall at all, and completions can be reaped without one either. What remains is the cost of publishing descriptors to memory and observing results through a shared tail pointer.

The shared region

io_uring_setup takes a queue depth and returns a file descriptor and a set of memory offsets. The descriptor is not a socket or a pipe; it identifies the ring context, and the offsets describe a single mapping that the application attaches with mmap. Inside the mapping sit the submission queue, the completion queue, a submission queue entry array, and a header of pointers and flags shared by both sides.

The submission side is an array of entries, each a fixed 64-byte structure, plus a ring of 32-bit indices into that array. The completion side is an array of result structures, each 16 bytes, with the same index ring. Both rings are circular and use a tail pointer written by the producing side and a head pointer written by the consuming side. Neither side reads the other’s pointer to decide whether a slot is free or full; each side computes the other’s index in the ring and compares against its own.

The important property of that layout is who writes what. The application writes entries into the submission array, advances the submission tail with a store that has release semantics, and the io_uring_enter call tells the kernel how many entries are now valid; the kernel then reads them directly rather than copying them in. The kernel writes completion entries and advances the completion tail; the application observes it and advances the head after consuming, with no call at all. A library wrapper such as liburing performs the tail update and submits from inside io_uring_submit, which is why the code below shows one call for a batch rather than one per entry.

struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fd, buf, len, offset);
sqe->user_data = tag;             /* recovered unchanged on completion */
io_uring_submit(&ring);           /* one enter for the whole batch */

/* Later, from the same or another thread: */
struct io_uring_cqe *cqe;
while (io_uring_peek_cqe(&ring, &cqe) == 0) {
    handle(cqe->user_data, cqe->res);  /* res is bytes moved or -errno */
    io_uring_cqe_seen(&ring, cqe);
}

Where the syscall comes back

Batching is what removes most of the transitions. io_uring_enter takes a number of submissions to consume and a minimum number of completions to wait for, so a program that queues a hundred operations and calls it once pays the fixed syscall cost once. With the wait count set to zero the call returns immediately after submitting, which allows a loop that submits and reaps in one place.

Polling removes the rest. When the ring is configured with a kernel polling thread, a thread of the kernel’s own checks the submission tail rather than waiting to be told about it. The cost moves from a privilege transition to a cache-line contention problem: the kernel’s polling thread reads a line the application writes, and that line bounces between cores. The trade is favourable when the application submits often and the work per submission is small, and unfavourable when the thread burns a core waiting on an idle application, which is why registration of the polling behaviour is optional and off by default.

There is a middle mode in which the kernel spins only for a bounded period at the end of a submission, so that the next request is picked up without another transition, and then sleeps. That mode makes the cost model a function of the arrival rate, which is closer to the real workload than either extreme. The setup flags that select among these modes are fixed at ring creation and cannot be switched afterwards, so the choice is a configuration decision rather than a per-call one. An application that alternates between idle and busy phases has to accept the polling thread’s unused cycles during idle, or pay transitions during busy periods.

Registered files and fixed buffers

The syscall path described in Where a read() Syscall Spends Time Before It Reaches the Data spends part of its time translating a descriptor number into a struct file and taking a reference, and part of it resolving the user buffer into pages. Both are avoidable if the resources are registered once.

A registered file table is an array; an entry in a submission queue entry can name a slot in that table instead of a descriptor number. The kernel resolves it without a descriptor-table lookup and without an atomic reference count. A registered buffer ring does the same for memory: registration pins the pages, records their addresses, and the request refers to the buffer by index. Registration costs an up-front pass over the memory, so it pays off when the same buffers and files are reused across many operations — which is the normal case for a network server or a database.

Registration also fixes a lifetime property. A pinned buffer cannot be reclaimed or migrated, so a program that registers a large amount of memory reduces the memory the kernel can use for other purposes. The page cache shrinks accordingly, which can slow unrelated buffered I/O. Registration is therefore a resource decision as much as a performance one, and the size should reflect the concurrency the application actually maintains, not its peak.

Linked operations and ordering

A single submission queue entry describes one operation, but operations can be linked: a flag on an entry makes the next one conditional on it, either on success only or on any completion. The chain runs in order internally. Chained operations can express a read followed by a write to another descriptor, or a fsync after a series of writes, without a kernel round trip between each step.

This is where the ring interface changes what a durably ordered program looks like. A log writer can submit a batch of writes and a flush in one chain and wait only for the last completion, which collapses what would be N round trips into the device’s own batching. The durability semantics of the flush are the ones described in What fsync Guarantees and What It Costs on Real Devices; the ring changes the number of transitions, not the guarantees.

Cancellation is the weak part of the model. A request that is already executing cannot always be stopped; a cancellation request can succeed, fail, or complete the original operation normally depending on where it is in the stack. A program that relies on cancellation for timeouts must be prepared for the original completion to arrive as well, which makes the timeout a suggestion rather than a guarantee. The alternative is to close the ring, which drains everything and is deterministic.

Failure modes

Entries that reference an unregistered index, a negative length, or an invalid operation return an error in the completion queue rather than as a syscall failure. The program that submits without synchronising therefore learns about its mistakes at reap time, and a loop that never reaps loses the errors. This shifts error handling from a return value into the completion loop, and forgetting to check the result field for every completion is a common way to drop failures silently.

Ring overflow is a second issue. The submission queue has a fixed capacity; entries written beyond it before the kernel consumes them are refused or, in some configurations, dropped. The application must check the space available before publishing, which is a read of the kernel-updated head pointer. A batch larger than the depth has to be split.

The completion queue can also fill. If the program submits more operations than it reaps, completions accumulate until the ring is full, and the kernel’s options are to allocate a larger overflow structure or to drop completions, which it reports. A program whose reap rate lags its submission rate will eventually hit one of these behaviours, so reaping must be a first-class part of the loop rather than an afterthought.

Behind all of this are the same device and block-layer mechanics as any other path. Depth is what the ring provides; the merging behaviour described in Writeback Scheduling and Merging at the Block Layer decides what the device sees, and the interface documentation of the filesystem decides whether an operation completes against the cache or the medium.