XDP and eBPF in the Receive Path

The kernel’s normal receive path allocates a socket buffer for each packet, fills in its metadata, and passes it up through the protocol layers. That allocation and the associated bookkeeping cost more than the decision most virtual networks need to make, which is often a lookup on a few header fields followed by a redirect. XDP moves the decision to before the allocation, on the raw frame, in a program the kernel verifies and then runs in the driver’s receive context. The redirect it performs is the same tunnel-encapsulation step that the overlay and underlay split describes at the protocol level, executed closer to the hardware.

The position in the path

An XDP program is attached to a network device and invoked for each received frame, before any socket buffer exists. Its input is a context holding a pointer to the start of the frame, a pointer to its end, a pointer to metadata the previous stage may have left, and the index of the interface the frame arrived on. The program may read and, for some cases, write the frame, and it returns one of a small set of verdicts.

XDP_PASS hands the frame to the normal stack, which then allocates the socket buffer and proceeds as if nothing had happened. XDP_DROP discards it, and the only trace is a counter. XDP_TX sends the frame back out the interface it arrived on, which is how a simple reflector is built. XDP_REDIRECT sends it to another interface or into a map of destinations, which is how forwarding between interfaces is built without the stack.

The verdict set is deliberately small, and each verdict has a cost. XDP_PASS pays the full stack cost after the program returns, so a program that passes everything has added a run of instructions to the existing path. XDP_REDIRECT avoids the stack entirely but requires the destination to be prepared: the frame must be transmitted from a device that can accept it, and the driver’s transmit path must be reachable from the receive context.

The return value is the entire interface between the program and the kernel. There is no way for a program to hand a packet to a later stage of the stack selectively: it may rewrite the frame and return XDP_PASS, but it cannot choose which protocol handler the stack then runs, and it cannot see the socket the packet would have reached. The design is a filter in front of the stack, not a replacement for it.

The three attach modes

The same program can run in three places, and the choice determines what it can see and how fast it runs.

Native mode attaches in the driver, so the program runs on the frame as the driver received it, before the driver builds a socket buffer. This is the fast path: the frame is already in a DMA buffer, the program operates on it in place, and the verdict can avoid the allocation entirely. It requires the driver to implement the hooks, and it is limited to the frames the driver’s receive ring delivers, which for some drivers means a single queue or a limited set of queue types.

Offloaded mode runs the program on the NIC itself, so the frame is processed before it reaches host memory. The capability is limited by what the hardware can execute, which is a small subset of the instruction set and a small set of maps, and by the hardware’s ability to parse the packet. It is the only mode that can drop traffic without spending host memory bandwidth, and it is the only mode whose performance does not depend on the host’s CPU.

Generic mode runs the program in the kernel’s generic receive path, after the socket buffer has been allocated. It works with any device, because it does not depend on driver support, and it costs what the stack costs plus the program. It exists for development and for devices where native mode is unavailable, and it is not a performance mechanism.

The practical consequence is that a program written for native mode may not run unchanged in offloaded mode, and a program written for generic mode may run in native mode with different results, because the frame layout the program sees differs. A program that assumes the frame starts at the beginning of the data and that the headroom is a particular size will work in one mode and not another. Portable programs compute offsets from the context rather than assuming them.

What a program can and cannot do

The verifier is the constraint that shapes every program. It performs a static analysis before the program is allowed to run, proving that every memory access is within bounds, that every loop terminates, and that no pointer is dereferenced after a check that does not dominate the use. A program that fails verification is rejected at load time, with a diagnostic naming the instruction.

The bounds proof is why packet access in XDP is written as a sequence of checks followed by a direct read, rather than as a structure dereference. The program must compare a pointer offset plus the size of the field against the data end before reading, and the verifier tracks that the comparison was made and that the pointer has not changed since. A read after a check that the verifier cannot see as dominating is rejected even if the program is correct.

That discipline is what makes the programs safe to run in the receive context, where there is no process to kill and no exception handler to recover. A wild read in a driver’s receive path would corrupt memory the kernel cannot attribute to anything, so the verifier’s conservatism is the price of running untrusted code at that level.

The program also cannot sleep, cannot allocate memory, and cannot call arbitrary kernel functions. It can call a fixed set of helpers, each of which is declared with its argument types and its effects, and it can access maps, which are the kernel data structures shared between the program and userspace. A map lookup is a helper call; which helpers and which map types an XDP program may use is restricted by the hook it is attached to, because the program runs in a context where blocking on a lock or on another CPU would stall the receive path rather than delay a task.

Maps as the state the program needs

A program that forwards between virtual networks needs to know which network a frame belongs to and where the destination is. That knowledge lives in maps, which are the only persistent state available.

The map types differ in what they can be keyed on and how they are looked up. A hash map is keyed on a byte array and supports arbitrary lookups, at the cost of a hash computation and a pointer chase. An array map is indexed by an integer and supports a lookup in constant time with no hashing, which makes it suitable for per-interface or per-queue state where the index is known. A longest-prefix-match map, usually implemented as an LPM trie, is keyed on a prefix and finds the most specific match, which is what a forwarding table needs.

The choice matters because the lookup happens per packet. An array lookup is a bounds check and a load; a hash lookup is a hash and a dependent load; an LPM lookup is a trie walk whose depth is the prefix length. A forwarding table with a million prefixes is a trie walk of up to thirty-two steps for IPv4, which is more work than a single hash but bounded and predictable.

The other half of the map story is how entries get there. A control plane running in userspace writes the entries through the bpf() system call, and the program reads them. There is no transaction across the two, so an entry written while the program is running is visible to the next packet, and an entry removed is invisible to the next packet. That is the same eventual-consistency model as any shared table, and it means a program must handle the case where a lookup fails even for a destination it expected to be present.

Where this fits in a virtual network

The natural use is the encapsulation boundary. A program attached to the interface facing the guests can, on the transmit side or the receive side, look up the destination in a map and either encapsulate and redirect or pass to the stack. The frame is already in memory, the header rewrite is a bounded set of writes, and the redirect avoids a socket buffer entirely for the traffic that is being tunneled.

The limits are the ones the position imposes. A program cannot terminate a TCP connection, because it has no access to the socket layer; it can drop, forward, or rewrite, but it cannot participate in the transport state machine. It cannot reassemble fragments, because it sees one frame at a time and has no reassembly context. And it cannot perform the kind of multi-table classification that a table-driven software switch provides, without building the equivalent structures itself, because the pipeline abstraction is not part of the XDP interface.

What it can do is replace the slow path for the cases that dominate. A program that decides encapsulation and output for the common case, and passes everything else to the stack, gets the benefit on the traffic that matters and keeps the stack’s full behaviour for the traffic that needs it. That division is the same one the pipeline design makes between its cache and its tables, expressed at a different layer of the system.