The Virtio Split Ring: Descriptor Tables, Available Rings, and the Notification Suppression

A device emulated in software charges an exit for every register write. A network adapter that transmits one packet per register write costs one exit per packet, and the exit handler must decode the write, find the queue, and build a descriptor. That is the wrong shape for I/O, where the guest may submit hundreds of requests in a burst and the hypervisor would rather handle them all in one visit.

Virtio replaces the register interface with memory. The guest and the hypervisor share a set of rings in guest memory, and a request is a descriptor the guest writes plus an index it advances. The hypervisor reads the index when it is notified, processes however many requests have accumulated, and advances a second index to say so. Notification is still required, but it is now decoupled from the request rate: one notification can cover an arbitrary number of requests.

Three structures with distinct owners

The virtio split ring, the original layout, has three parts. Each has exactly one writer, and that ownership rule is the whole design.

The descriptor table is an array of fixed-size entries. Each entry holds a guest-physical address, a length, and two flags: whether the buffer is device-readable or device-writable, and whether another descriptor follows. The table is shared, but the guest owns allocation of entries and the device only follows the chains the guest builds. A request can span several descriptors, chained by the next flag, which is how a network packet with a header and payload in separate buffers is described without copying either into one contiguous region.

The available ring is written by the guest. It holds the indices of descriptor chains the guest has made ready, in the order the guest wants them processed, and a counter of how many entries have been added. The guest adds the head index of each chain, increments the counter, and notifies the device.

The used ring is written by the device. It holds the head index of each completed chain and the number of bytes the device wrote into it. The guest reads it to reclaim descriptors.

The separation into two rings rather than one is what allows both sides to make progress without locks. A single ring with a head and a tail pointer would need the two sides to update the same cache line, and the cache line would ping-pong between them. Two rings put the guest’s writes and the device’s writes on different lines, and the counters that both read are the only shared mutable state.

/* Guest side: publish two descriptors as one chain, then advance the index. */
static void submit(struct virtq *q, uint16_t head, uint16_t avail_idx)
{
    /* Descriptor writes must be visible before the index that refers to them. */
    __sync_synchronize();

    q->avail->ring[avail_idx % q->num] = head;

    /* Publish the new count; the device reads it to learn what is ready. */
    __sync_synchronize();
    q->avail->idx = avail_idx + 1;

    /* Skip the notification if the device asked for suppression. */
    if (!(q->avail->flags & VRING_AVAIL_F_NO_INTERRUPT))
        notify_device(q);
}

The memory barriers are the contract

Both sides write data and then write an index that makes the data reachable. The processor and the compiler are free to reorder those writes, and the device may be reading the memory through a path that does not observe them in program order. Every index update is therefore preceded by a barrier that orders the data writes before the index write, and followed on the reader’s side by a barrier that orders the index read before the data reads.

The specification states this explicitly rather than leaving it to the implementer, because getting it wrong produces a failure that appears only under load and only on some architectures. A device that reads a descriptor before the guest’s write to it is visible will transmit whatever the buffer previously held, which is a data corruption bug that no amount of testing on a strongly ordered machine will reveal.

The direction matters too. The device writes completion information into the used ring and into the buffers, then updates the used index. The guest reads the index, then reads the entries. Without the barrier pair, the guest can observe an incremented index and then read a completion entry that has not been written yet.

Notification suppression, in both directions

The naive design sends a notification for every request and receives an interrupt for every completion. That is a trap per request in each direction, which defeats the purpose. Virtio provides suppression in both directions, and the two mechanisms are asymmetric.

The available ring’s flags field lets the device tell the guest that it does not need a notification for the next batch. If the flag is set, the guest may add descriptors without notifying, because the device has committed to checking the ring on its own schedule. This is a promise about future behaviour, and a device that sets the flag and then never polls will starve. Implementations therefore set it only when they have another reason to look at the ring, such as a timer or a batch boundary.

The used ring’s flags field is the same mechanism in reverse, letting the guest tell the device not to interrupt on completion. The guest sets it when it knows it will poll the used ring anyway, typically while it is already processing a batch of completions. A driver that never clears the flag will not receive interrupts at all, which is correct if it polls and wrong if it does not.

A third control, the event index, refines the same idea. The guest writes an index into a shared location saying “interrupt me only when the used index reaches this value”. The device compares its own progress against that value and skips the interrupt if it has not reached it. This replaces the coarse flag with a precise threshold, at the cost of one more shared cache line and one more read on the device’s completion path.

The effect of all three is that notification count becomes a tunable independent of request count. A guest that submits a thousand requests and suppresses notifications until the batch is complete pays one interrupt. A guest that needs low latency submits one request and notifies immediately. The interface does not force either choice.

Descriptor chains and the cost of indirection

A descriptor chain is a linked list in an array, followed by index rather than by pointer. The device reads the head descriptor, follows the next flag to the next index, and stops at the last. Each hop is a memory read of a sixteen-byte entry, so a chain of three buffers costs three reads of the table plus one read per buffer.

The chain length is bounded by the queue size, which is a power of two between 2 and 32768 entries in the split ring layout, and the guest allocates the table. A longer chain means more per-request overhead; a shorter chain means the guest must copy data into contiguous buffers. The trade is the classic scatter-gather trade, and virtio lets the guest decide per request.

The indirection has a security dimension. The device trusts the guest’s addresses, but the addresses are guest-physical and the device must translate them through the second-level tables. A malicious guest can build a chain that loops, by pointing a descriptor’s next field back at an earlier index. The device must bound the number of descriptors it follows, and the specification requires it: a chain longer than the queue size is invalid. Without that bound, a loop in the descriptor table hangs the device model, and since the device model often runs in the hypervisor’s address space, it hangs the host.

Packed rings and why the split layout was replaced

The split ring’s three structures require the guest to allocate and initialize three separate arrays, and the indices wrap at the queue size, so every access needs a modulo. The packed ring layout, added in a later version of the specification, puts descriptors and the available and used information into a single array, with each descriptor carrying the flags that say whether it is available or used.

The packed layout reduces the memory footprint and removes the separate ring arrays, and it changes the notification rules: the flags live in the descriptor itself, so suppression is per-descriptor rather than per-ring. The cost is that the descriptor is now written by both sides at different times, so the ownership rules become temporal rather than spatial, and the barrier requirements become more delicate.

Both layouts coexist, and a device negotiates which one the driver will use. A hypervisor that supports only the split ring cannot offer the packed features, and a driver that supports both carries two code paths for the same queue.

Where the ring ends and the device begins

The ring carries requests, not semantics. A block device’s request is a descriptor chain whose first buffer holds a header naming the operation, the sector, and the length, and whose subsequent buffers hold the data. A network device’s request is a chain whose first buffer holds a header the device fills with metadata and whose second holds the frame. The ring does not know either format.

That separation is why one transport serves many device types, and it is also why the device-specific headers are where the compatibility problems live. A header whose layout changes between versions of the specification forces the driver to negotiate a feature bit, and a device that claims the feature without implementing the layout produces a driver that reads fields at the wrong offsets. The ring is stable; the structures it carries are the moving part.

The hypercall interface supplies the notification mechanism, and the ring supplies the data path. Neither is useful alone: a ring with no notification is polled and wastes cycles, and a notification with no ring is a trap per request. The pairing is the actual contribution of paravirtualized I/O, and it is why the model survived the arrival of hardware that can intercept device access directly.