The Virtio Split Ring: Descriptor Chaining and the Two Suppression Flags

A paravirtual device’s whole advantage is that the data path never crosses the hypervisor boundary. Buffers live in guest memory, the device reads them by DMA, and the only signals exchanged are a doorbell write in one direction and an interrupt in the other. The ring that makes this work is a small, rigidly specified data structure, and its layout is designed so that both sides can operate on it with single-word atomic operations and no locks.

Three arrays in one contiguous region

The split virtqueue occupies one region of guest-physical memory containing three arrays, at fixed offsets from a base address the driver chooses and reports through a configuration field.

The descriptor table comes first, at offset zero. It is an array of 16-byte entries, one per queue slot. Each entry is four fields in little-endian order: a 64-bit address, a 32-bit length, a 16-bit flags word, and a 16-bit next index. The address is a guest-physical address, because the device performs DMA and has no knowledge of guest-virtual mappings. The length is the buffer size. The flags word uses bit 0 for “next descriptor follows”, bit 1 for “device writes into this buffer”, and bit 2 for “this descriptor points to an indirect table”. The next field is meaningless unless the first flag is set.

The available ring follows at offset 16 × queue_size. Its header is a 16-bit flags word and a 16-bit index, then an array of queue_size 16-bit entries, then, if the event-index feature is negotiated, one more 16-bit field. Each ring entry holds the index of the head descriptor of a chain the driver is offering. The driver’s index counts total chains offered since the queue was reset, wrapping modulo 65536; it is not the position in the array, which is the index modulo the queue size.

The used ring follows at the smallest even offset at or after 16 × queue_size + 6 + 2 × queue_size. Its header is the same shape, but its entries are 8 bytes each: a 32-bit descriptor index identifying which chain completed, and a 32-bit length giving the number of bytes written. The used ring carries the device’s index in the same wrapping form. The extra six bytes in that offset are the available ring’s flags word, index word, and the two bytes reserved for its event field, which are accounted for in the layout whether or not the feature that gives them meaning is negotiated.

Queue size must be a power of two, and the descriptor table must be 16-byte aligned. Both constraints come from the access patterns: the power of two lets both sides compute the array slot with a mask instead of a division, and the alignment lets a single 16-byte store write one descriptor without a read-modify-write.

struct virtq_desc {
    uint64_t addr;
    uint32_t len;
    uint16_t flags;
    uint16_t next;
};

struct virtq_avail {
    uint16_t flags;
    uint16_t idx;
    uint16_t ring[];      /* queue_size entries, then used_event */
};

struct virtq_used_elem {
    uint32_t id;
    uint32_t len;
};

struct virtq_used {
    uint16_t flags;
    uint16_t idx;
    struct virtq_used_elem ring[];
};

Chaining and the indirect table

A chain is a linked list through the descriptor table. The driver writes descriptors for each buffer, links them with the next field, and places the head index in the available ring. The device walks the chain, performing a transfer for each descriptor, and stops when it reaches one without the next flag.

Chains exist so that a single logical request can span non-contiguous buffers. A network transmit packet has a header buffer and a payload buffer that the guest’s stack produced separately; chaining lets the device read both under one available-ring entry, so the completion is one used-ring entry rather than two. For a receive queue, the driver typically posts chains of two: a small header buffer the device writes, and a larger data buffer.

Indirect descriptors flatten the chain. When the flag is set, the descriptor’s address points not to a data buffer but to a table of descriptors in guest memory, and its length gives the table’s size in bytes. The device follows the table as if it were part of the descriptor table. The benefit is queue capacity: a chain of many buffers consumes many descriptor slots, while an indirect chain consumes one. The cost is an extra dependent memory read before any data transfer begins, which matters when the table is not in cache.

The specification bounds indirect tables so that a device can allocate a fixed-size buffer to copy one into: the table may hold at most as many entries as the queue size, and the device may limit the number of descriptors it will follow in a single chain.

Suppression in both directions

Each side can signal the other, and each signal is expensive. A doorbell write from the driver is an MMIO exit into the hypervisor. An interrupt from the device is an injected interrupt into the guest. Both are worth suppressing when the other side is already awake.

The original mechanism is a pair of flag bits. The driver sets a bit in the available ring’s flags word to tell the device not to interrupt on completion of the chains it is about to add. The device sets a bit in the used ring’s flags word to tell the driver not to kick the doorbell when it adds buffers. The bits are advisory and racy: the driver sets the flag based on its belief about the device’s state, the device may have already decided to interrupt, and the interrupt arrives anyway. Waking up to find no work is harmless but wasteful, and the race means the flags cannot guarantee suppression.

Event index replaces the flags with counters. When the feature is negotiated, the available ring gains a used_event field and the used ring gains an avail_event field, and both sides switch from testing a bit to comparing an index.

The driver writes into used_event the value of the used index at which it next wants an interrupt. It then adds buffers and, before doing so, reads the used ring’s current index. If the device has already passed the point the driver cares about, no interrupt is needed. The device, before adding a completion, compares its new used index against the driver’s used_event value using the standard wrapping comparison: the device interrupts when the difference between the new index and the event value, taken modulo 65536, is less than the number of entries it just added. The same comparison in the other direction governs avail_event and the doorbell.

This is exact rather than advisory, and it collapses the interrupt rate to the driver’s actual consumption rate. A driver that processes sixteen completions per interrupt sets used_event sixteen ahead and receives one interrupt per sixteen completions, with no spurious wake-ups even under races. The cost is that both sides must use the wrapping comparison correctly; a naive integer comparison produces either a storm of interrupts or none at all when the index crosses 65535.

/* Should the device interrupt after advancing used->idx by `added`? */
static int vring_need_event(uint16_t event_idx, uint16_t new_idx, uint16_t old_idx)
{
    /* wrapping comparison: is event_idx in (old_idx, new_idx]? */
    return (uint16_t)(new_idx - event_idx - 1) < (uint16_t)(new_idx - old_idx);
}

Modern transport and the doorbell

The split ring is transport-independent, but the notification path is defined by the transport. Under the PCI transport, the driver locates the notification address through a device capability, and writes the 16-bit queue index to it to kick that queue. A further feature allows a 32-bit write whose value is a per-queue notification data word, so a device can map several queues to distinct addresses and avoid the index lookup.

Under the MMIO transport, the notification is a write to a register with the queue index as the value. Either way, the write is the expensive part, and the number of writes is what event index controls. A driver that kicks once per descriptor will saturate the exit path no matter how the device is implemented; a driver that fills the ring before kicking pays one exit per ring.

The receiving side has a corresponding cost profile. An interrupt delivered to a guest running with posted interrupts costs no exit, but it still costs a trip through the guest’s interrupt handler, and the guest’s handler must then walk the used ring to find the completed chains. Suppressing to a batch means each interrupt amortizes that walk over more completions.

Failure modes at the boundary

The ring’s correctness depends on memory ordering that neither side can enforce alone. The driver must ensure that descriptor writes and the available-ring entry are visible to the device before the index update is visible, because the device may read the index, then the entry, then the descriptor. On x86 the store ordering provides this; on weaker architectures the driver needs an explicit barrier before publishing the index, and the specification requires one. A driver that publishes the index first lets the device read a descriptor that is not yet written, which produces a transfer from stale or uninitialized memory rather than an error.

The reverse ordering applies to the device’s completion: it writes the used entry, then publishes the index, then signals. The driver reads the index, then the entry. Without the barrier, the driver can observe a completion for a descriptor whose length field is not yet written.

A second failure mode is descriptor reuse. The device returns a descriptor index in the used ring, and the driver may not reuse that chain until it has seen the completion. A driver that reuses too early overwrites a buffer the device is still writing, which corrupts data silently. The used index is the only signal that a chain is free, and its wrapping nature means the driver must track how many chains are outstanding rather than trusting the raw value.

A third is the zero-length descriptor. The specification permits it, and a device that treats a zero length as an error will fail on drivers that use it as a placeholder. The behaviour a device must implement is a transfer of zero bytes that still counts as a completion, which is the sort of detail that separates a conforming implementation from one that works with the drivers it was tested against.

Where the ring ends and the hypervisor begins

The ring is guest memory, so the device’s access to it is DMA, and the hypervisor’s involvement is limited to translating the guest-physical addresses in the descriptors and pinning the pages so they stay mapped. That translation is the subject of the copy-counting analysis of the virtual I/O path, where the choice between a userspace device, an in-kernel device, and a passthrough device determines how many times the payload is touched between the guest’s buffer and the wire.