Counting Copies on the Virtual I/O Path: Userspace, vhost, and Passthrough

A guest transmitting a packet performs one copy: from its socket buffer into the transmit ring. Everything after that is the hypervisor’s problem, and the number of additional copies depends entirely on where the device model runs and how it reaches the network. Counting those copies is the clearest way to compare the three common device models, because each removed copy corresponds to a specific mechanism and a specific loss of isolation.

The baseline: a device model in a userspace process

The first implementation of virtio put the device model in the hypervisor’s userspace process, alongside the emulated chipset and firmware. For a transmit, the driver fills the ring and kicks. The kick is an MMIO exit, which the kernel’s exit handler recognizes as belonging to this device and forwards to the userspace process, typically by signalling an event file descriptor that the process is polling.

The userspace model then reads the ring from guest memory. It must translate the guest-physical addresses in the descriptors into host-virtual addresses it can dereference, which on Linux means a lookup in the mapping table the hypervisor maintains for the guest’s memory slots. That lookup is a hash or interval-tree search, and it must be done per buffer, though implementations cache the last slot. Once translated, the model can read the descriptor’s buffer directly, because the guest’s memory is mapped into the process.

So far the model has read the payload once, into its own address space, without copying. The copy appears when the data must leave: the process hands the buffer to the host network stack, which expects a kernel buffer, so the payload is copied into a kernel socket buffer. If the host stack then forwards it to a physical NIC, the NIC’s driver may copy again into a DMA-visible buffer, unless the stack has managed to keep the same pages.

The count for this path is two to three copies after the guest’s own, plus the exit and the process wake-up. The process wake-up is often the dominant cost: a context switch, a poll loop iteration, and a return, all for one packet unless the model batches.

Moving the model into the kernel

The obvious optimization is to remove the process boundary. A kernel-side device model, generically a vhost-style implementation, runs in the host kernel and is reached from the guest’s exit without a context switch. The kernel already has the guest’s memory mapped, so the translation is a direct lookup in a data structure rather than a message to another process.

The guest-to-host signal becomes cheaper in a specific way: the exit handler can call the device code directly on the same stack, with the vCPU’s context still hot. The wake-up disappears. What does not disappear is the translation of descriptor addresses and the pinning of the pages they refer to, because the device must guarantee that the memory it DMAs into does not move or get unmapped while the transfer is in flight.

The copy profile improves because the kernel model can pass the guest’s pages directly into the host networking path as a scatter-gather list, rather than assembling a linear buffer. If the host stack and the NIC driver both support it, the payload is never copied; the NIC DMAs from the guest’s pages. The mechanism that makes this legal is the IOMMU, which translates the guest-physical address the NIC is given into the host-physical address of the guest’s page, so the device can be programmed with an address it does not own and still reach the right memory.

Without an IOMMU, the host must either copy into a buffer the device can address, or program the device with the host-physical address obtained from the translation, which works but requires the pages to be pinned and contiguous enough for the device’s scatter-gather limits. The IOMMU turns a correctness requirement into a performance enabler, which is why its presence changes the achievable copy count.

Passthrough and the disappearance of the model

Assigning a physical device to the guest removes the model entirely. The guest driver programs the device directly, the device DMAs to guest-physical addresses translated by the IOMMU, and interrupts are routed to the guest. No exit occurs on the data path at all, and the copy count is whatever the guest driver and the device require, identical to bare metal.

The price is paid in flexibility. The device belongs to one guest. The host cannot inspect the traffic. Live migration requires either a device that can save and restore its state, or a mechanism that pauses the guest, detaches the device, and resumes it on the destination, which is a visible interruption rather than a move the guest does not notice. The host also loses the ability to interpose on the device’s behaviour, which is sometimes the point and sometimes the problem.

Scalable I/O virtualization occupies a middle position. A device exposes multiple virtual functions, each assignable to a different guest, with the hardware partitioning the queues and the interrupt resources. The data path is passthrough for each guest, and the management path remains with the host. This is the configuration that gives near-native I/O to many guests at once, at the cost of requiring hardware that implements the partitioning and a host driver that manages the functions.

What each copy actually costs

A copy of a 1500-byte packet is not a single operation. Reading the source pulls cache lines; writing the destination may require a read-for-ownership if the destination line is not already in the local cache, because the write covers only part of a 64-byte line. On a typical server, a memcpy of that size is tens of nanoseconds when both buffers are hot and several times that when they are not. At a million packets per second, the difference between one copy and three is a core’s worth of capacity.

The arithmetic is why zero-copy is pursued, and also why it disappoints. Removing a copy removes the memcpy but adds bookkeeping: reference counts on pages, completion callbacks that must fire after the device is done with the memory, and a mechanism to delay the guest’s buffer reuse until the DMA has completed. The guest’s driver has already promised not to reuse the buffer until the used ring reports completion, so the host’s bookkeeping must respect the same boundary, and a host that returns the buffer early corrupts the guest’s data.

There is also a cache effect that copies hide. A copied packet warms the destination buffer in the local cache, so the NIC driver’s descriptor setup and the host stack’s header parsing hit in cache. A zero-copy path hands the NIC pages that the guest recently wrote, which are in the guest’s cache footprint; if the guest is running on a different physical core, those lines may be in another cache and require transfer. Zero-copy removes bandwidth but not necessarily latency.

The exit is still there

None of the device models described here remove the guest’s doorbell. The driver still writes the queue index to the notification address, and that write is still an MMIO exit into the hypervisor, which then dispatches to whichever model is in use. For a passthrough device the notification address is in the device’s own BAR, so the write goes to hardware and no exit occurs; for a virtio device it is in the emulated transport’s BAR, so it exits.

The mitigation is the one described in the ring’s suppression mechanism: batch the ring, kick once. The device model’s cost per packet then falls to the per-descriptor work, and the exit is amortized. A driver that kicks per packet cannot benefit from any of the copy elimination described above, because the exit dominates.

The same asymmetry applies on the receive side. A guest’s receive queue is filled with buffers before the device needs them, and the device writes into them by DMA. If the device model is in userspace, it must translate the buffer addresses once when the buffers are posted, not per packet, and the actual receive is a DMA into guest memory with no copy at all. The interrupt that reports the completion is the cost, and posted interrupts remove the exit for that.

Choosing a model

The decision is not purely about copies. A userspace model can be restarted, debugged, and instrumented without touching the host kernel, and a bug in it does not panic the host. A kernel model has a smaller trusted computing base for the guest’s data path but a larger one for the host’s stability. Passthrough has the smallest software path and the largest hardware dependency.

Device model	Host copies of guest payload	Per-packet host cost	Isolation and flexibility
Userspace model behind the hypervisor	One to two, plus a context switch per batch	High; dominated by the wake-up	Model is a user process, restartable and inspectable
Kernel-side model	Zero to one, dependent on host stack and IOMMU	Low; exit handled on the vCPU stack	Code runs in the host kernel
Passthrough through an IOMMU	None	Lowest; the device decodes the access	Device belongs to one guest, migration is hard
Hardware-partitioned virtual functions	None	Lowest per guest, shared management	Partitioned in hardware, host manages the functions

The copy count maps onto that trade reasonably well. Userspace models pay copies and context switches; kernel models pay pinning and IOMMU programming; passthrough pays inflexibility. A workload that moves a lot of bytes through few large transfers will not notice the difference, because the per-byte cost dominates. A workload that moves many small messages will notice every one of them, because the per-operation cost dominates and the copies are the smallest part of it.

The remaining fixed cost in every model is the translation of guest addresses. Even with passthrough, the IOMMU performs a walk per access that misses its cache, which is the same two-level structure as nested paging, and the same advice applies: the size of the working set and the granularity of the mappings decide whether that walk is a cache hit or a memory round trip.