Inside the NIC Receive Path: Descriptors, Rings, and NAPI

Between the wire and the protocol stack there is a queue of descriptors, a direct-memory-access engine, and a decision about when to tell the processor. A frame that is dropped anywhere in that machinery never reaches a counter that the protocol stack knows about, which is why a host can report no errors and still lose traffic. The receive path is worth understanding in its own right because every one of its design choices is a latency-versus-throughput trade that shows up as a specific symptom.

Descriptors and the ring

The driver allocates a region of memory, the descriptor ring, and divides it into fixed-size slots. Each slot describes one buffer: a physical address, a length, and status bits. The NIC holds the ring’s base address and a register pointing at the slot it will fill next. For each arriving frame, the NIC’s DMA engine writes the frame bytes into the buffer the slot points at, updates the slot’s status to mark it complete and record the frame length, and advances its pointer. No processor involvement is needed until the ring has something in it.

The ring is a ring because the driver and the hardware must agree on position without a shared counter. The NIC increments its index and wraps at the end; the driver consumes slots and returns them to the NIC by clearing the ownership bit. A slot that the driver has not returned still belongs to the hardware, so the NIC will not overwrite a buffer the stack is reading. The direction of ownership is encoded in one bit of the descriptor, and the driver’s job is to flip it only after the frame has been fully consumed.

The buffer sizes are a separate decision from the descriptor count. A buffer must be at least as large as the largest frame the interface accepts, so a 1500-byte MTU needs buffers of at least 2 KB to hold the frame plus alignment and any header room the stack wants. A jumbo-capable interface needs buffers sized for 9000 bytes or more, which changes the memory footprint of the ring by a factor of four and makes the ring less effective at absorbing bursts, since the same memory holds fewer buffers.

Interrupts, and why one per frame is wrong

The naive design has the NIC raise an interrupt for every completed frame. At low rates that is the lowest-latency option: the frame is handed to the stack as soon as it arrives. At high rates it collapses. A 10 Gbit/s link carrying minimum-size frames delivers roughly 15 million frames per second counting the preamble and interframe gap, and no processor can take 15 million interrupts per second and still do the work of processing them. The interrupt overhead alone, at hundreds of cycles per entry and exit, exceeds the available cycles.

Two mechanisms address this, and they are complementary. The first is interrupt coalescing: the NIC waits until either a number of frames have accumulated or a timer has expired, then raises one interrupt for the batch. The parameters are the frame count and the timer, both configurable, and the trade is direct: a larger count amortises the interrupt further and adds latency, because the first frame of a batch waits for the rest.

The second is to stop using interrupts as the primary mechanism at all. The kernel’s approach, NAPI, is that the first interrupt disables further interrupts from that interface and schedules a poll. The poll runs in softirq context with a budget: it processes up to a fixed number of frames or a fixed time slice, and if the ring still has work when the budget is exhausted, it reschedules itself and continues. When the ring empties, interrupts are re-enabled. Under load the interface stays in polling mode continuously, which is where the throughput comes from; when traffic stops, the interface returns to interrupt mode, which is where the latency comes from.

The budget is what prevents a single interface from monopolising a CPU. A poll that processed the entire ring could starve every other softirq on the same CPU, including the transmit completions and the timer processing. The budget bounds that, at the cost of a reschedule, which is a softirq re-entry rather than a hardware interrupt and is much cheaper.

The queues between the ring and the stack

A completed descriptor does not immediately become a packet for the stack. The driver builds a socket buffer structure that points at the DMA buffer, records the frame length, and attaches metadata: which interface, which receive queue, the checksum status if the hardware computed one, and the VLAN tag if the hardware extracted it.

The checksum status is worth attention because it is where offload changes the contract. When receive checksum offload is enabled, the NIC computes the transport checksum and reports in the descriptor whether it was correct. The driver marks the packet as checksum-verified, and the stack skips its own computation. If the NIC reports a bad checksum, the packet is dropped before it reaches a socket, and the drop is counted separately from a CRC error, because the frame itself was intact; the payload was corrupt in a way the frame check sequence did not catch.

The transmit side has the mirror image: with large send offload, the stack hands the NIC a buffer larger than the MTU and the NIC splits it into frames. On receive, large receive offload coalesces several frames of one flow in hardware, so the stack sees one packet with a larger payload instead of several. That reduces per-packet overhead but destroys the original frame boundaries, which is why interfaces used for forwarding disable LRO: a bridge or router must reproduce the frames it received, and the boundaries are no longer available. Generic receive offload performs the coalescing in software but keeps enough metadata to segment the packet again before it is forwarded.

Where frames are lost

Every stage has a bounded capacity, and exceeding it loses frames silently at the link layer:

Stage	Bound	Symptom
NIC receive FIFO	Hardware buffer, often tens of KB	Missed error counter increments
Descriptor ring	Number of slots the driver allocated	No buffer counter, then missed
Poll budget	Frames per softirq invocation	Backlog grows; drops appear later
Socket receive queue	Per-socket limit	Drops reported at the socket

The first is the least visible. A frame arrives when the NIC’s internal FIFO is full, and the NIC discards it before any descriptor is involved. The counter for this is separate from the ring’s, and on many drivers it is exposed under a name that does not obviously mean “the hardware had nowhere to put it”. The ring’s version is easier to reason about: if the driver has not returned descriptors fast enough, the NIC has no slot to write into, and the frame is dropped with a counter that increments per event rather than per frame on some hardware.

The poll budget’s effect is indirect. A budget that is too small relative to the arrival rate means the backlog never drains, the ring fills, and frames are lost at the ring rather than at the poll. The observable signature is a ring-full counter rising while the CPU running the poll appears only partly busy, because the poll is being rescheduled rather than run continuously.

Multiple queues and where a frame is steered

A modern NIC has many receive queues, and each can be polled independently, possibly on a different CPU. The steering decision is made by the hardware from a hash over the flow’s fields, using the same kind of hash as link aggregation, and the queue index is derived from it. The purpose is to keep a flow on one CPU so that its packets are processed in order and its cache lines stay warm.

The steering is not perfect. A hash collision puts two flows on one queue while another queue is idle, and a flow whose headers differ from packet to packet can move between queues: IPv4 fragments after the first carry no transport header, so the hash reads different fields and can place them elsewhere, and the stack’s per-CPU processing then reorders them. The usual approach is to receive on the queue the hardware chose and to let the stack’s flow steering decide the rest; steering every packet in software instead adds per-packet cost.

The relationship between the queue count and the CPU count matters. More queues than CPUs means some CPUs poll several queues and the per-CPU serialisation that the steering was meant to provide is lost. Fewer queues than CPUs leaves CPUs idle while others are saturated. The configuration is a per-interface parameter, and it interacts with the interrupt affinity settings, since a queue’s interrupt must be delivered to the CPU that polls it.

Why the path is structured this way

The alternative to a descriptor ring is programmed I/O, where the processor reads each frame from the device’s registers. That is simpler and it is what small embedded controllers do, but it costs a memory-mapped read per word and cannot approach wire rate. DMA exists to move the bytes without the processor, and the descriptor ring exists so that DMA has a list of destinations without the processor writing a register per frame.

The alternative to polling is pure interrupts, which loses at high rates, or pure polling, which burns a CPU when idle. NAPI’s hybrid is the compromise: the mechanism that is efficient under load is used under load, and the mechanism that is efficient when idle is used when idle. The transition between them is the interesting part, and it is why a latency measurement on an idle interface differs from the same measurement on a busy one.

Everything above this path assumes the frame arrived intact and was addressed to this station. The check that establishes the first is the trailer’s checksum, described in the frame check sequence, and the filter that establishes the second is described in MAC address structure and group addressing. A frame larger than the buffers the ring was sized for has nowhere to go and is dropped before the stack sees it, which is one mechanism behind the silent loss described in link MTU and jumbo frames.