Virtualizing Accelerators Without Hardware Partitioning

A hypervisor that gives a virtual machine a device has four strategies available, and they differ by where the device’s state lives. Expose a fully emulated device and every access is trapped. Hand the device to one machine and the others get nothing. Place a physical device behind a virtual one and let the guest drive it under supervision. Intercept the guest’s high-level API and forward the work to a shared device. Each answers a different question about isolation, and each is the right answer for a different kind of device.

Emulation, and where the exits accumulate

Full emulation presents a virtual device whose every register is implemented in the hypervisor. A guest store to a register causes a trap, the hypervisor decodes the instruction, discovers which register and value, updates its model, and resumes the guest. A guest read from the device is the same path in reverse.

This works without any hardware support and provides complete isolation: the guest cannot reach the physical device at all. Its cost is the trap. A trap-and-emulate cycle on the order of a thousand cycles per access, with the exact number depending on the virtualization extension and on how much state the guest was using. A driver that programs a device with a hundred register writes at initialization pays a hundred thousand cycles, which is tolerable. A driver that writes a doorbell per descriptor pays it per descriptor, which is not.

The mitigation is to batch. A virtual device designed for emulation will offer a doorbell that reports a count or an index rather than one per operation, so the trap rate scales with batches rather than with work. This is not an accident of design: the virtio family of virtual devices was shaped by exactly this constraint, and its queue format — a descriptor ring with an available index and a used index the driver polls — exists so that the number of traps per batch is two, one to submit and one to discover completion, rather than proportional to the number of buffers.

Emulation also has to model completion notification. The virtual device cannot interrupt directly, so the hypervisor injects an interrupt into the guest when the model decides one is due, which means the hypervisor is in the path of every notification and its own scheduling delays become the guest’s latency. A guest that polls its completion ring avoids the injection entirely, which is why the split-ring design is polled on the used half as well as posted on the available half.

Mediated passthrough

The middle ground gives the guest a real device’s registers but intercepts the operations that must be mediated. Address space assignment, interrupt configuration, and DMA mapping are handled by the hypervisor, while ordinary register reads and writes go straight through to hardware.

The advantage is that the fast path no longer traps. The guest driver’s doorbell write reaches the physical device without an exit, and the completion path is likewise direct. The cost is that the device must be shareable in the first place, and the mediation must be able to enforce isolation where the hardware does not.

The critical requirement is that the device’s DMA be confined to the guest’s memory. A mediated passthrough device is programmed by the guest with addresses the guest chooses, and the hypervisor must intercept those before they reach the device, translating them into machine addresses that belong to the guest and rejecting anything else. This is exactly an IOMMU translated domain per guest, described in Device DMA and the IOMMU Boundary, and it is a prerequisite rather than an optimization: without it, a guest driver can program a DMA to any physical address in the machine.

The second requirement is interrupt isolation. The device’s interrupt must be associated with the guest that owns it, and the hypervisor must be able to route it without involving the other guests. Otherwise a single shared interrupt line forces the hypervisor into the completion path, and mediated passthrough degrades toward emulation.

Mediated passthrough is usually combined with a small emulated control path: the guest sees a standard configuration space for the device class, and the hypervisor intercepts configuration accesses while leaving the memory-mapped register window mostly unmapped from its perspective. The split is chosen so that the trapped path runs once per setup and the untrapped path runs per unit of work.

API remoting

The fourth strategy does not virtualize the device at all. It intercepts the guest’s calls into its acceleration API and forwards them to a service outside the guest, which owns the device and executes the work on behalf of every guest.

This is attractive because it requires no device cooperation and no virtualization support in the device. It is also the most expensive of the four in the general case, because a call may carry arbitrary data: a buffer to read, a buffer to write, a structure to traverse. Every such call must be marshalled across the boundary, and the cost is proportional to the data touched rather than to the number of calls. Kernels that touch large buffers therefore generate large transfers just to describe themselves.

The technique becomes viable when the API is designed for it. An API whose command stream is a well-defined buffer that the guest fills and submits mostly writes — commands with buffer addresses and lengths, not buffer contents — can be forwarded by transferring the command stream and letting the device read the buffers directly through mappings the host establishes. This is where shared virtual addressing earns its keep: if the device can use the guest’s page tables, the host can hand it the guest’s pointers and skip the copy, at the price of the fault and teardown machinery described in Shared Virtual Addressing and Peer-to-Peer Transfers.

API remoting also changes the failure model. A device error surfaces as an error in the remote service rather than as a device error in the guest, so the guest’s driver sees a call that failed rather than a device that fell off the bus. Whether that is better or worse depends on whether recovery is expected to happen inside the guest or outside it.

Why SR-IOV is the answer when the device can be cut up

The hardware solution is to make one physical device present itself as many. A physical function manages the device and exposes a set of virtual functions, each of which is a complete PCI function with its own configuration space, its own base address registers, and its own interrupt. The hypervisor assigns a virtual function to a guest and the guest drives it with the device’s ordinary driver, with no mediation in the data path.

The isolation requirements are the same three: a per-function address space enforced by the IOMMU, a per-function interrupt, and a per-function queue set in the hardware. The difference is that all three are enforced by hardware rather than by the hypervisor, so the guest’s fast path contains no exits. A virtual function with its own address space cannot reach another guest’s memory even if the guest’s driver is malicious, and the hypervisor is not in the path to check.

The limit is that the device must be partitionable. A network controller with many independent queues partitions cleanly. A fixed-function engine with a single control register and a single execution pipeline does not: it has one state machine, one queue, and one set of counters, and there is no sensible way to present two of it. Devices with shared internal resources — a common cache, a shared memory controller, a single encryption pipeline — can be partitioned only if the resource is time-multiplexed and the multiplexing is visible to the hardware.

Choosing among them

The decision follows from three questions.

Does the device partition at hardware level? If it does, and the platform supports the required isolation, direct assignment through virtual functions is the only option with no data-path overhead.

Is the access pattern dominated by a small control path and a large data path? If so, mediated passthrough keeps the data path direct and pays the exit only at setup, which is the best of the software options.

Is the device’s interface a command stream whose contents are addresses and lengths rather than data? If so, API remoting can forward the stream cheaply, and shared addressing can remove the remaining copies.

If none of these hold, emulation is what remains, and its cost is the trap rate times the access frequency. For a device designed to be shared this is acceptable; for a device designed for direct programming by one driver it is not, and the practical outcome is that such a device cannot be shared at all except by giving it to one guest and letting the others go without.

The last consideration is the one that decides most deployments: the overhead must be weighed against what the guest actually does with the device. A guest that runs a device for a few milliseconds at startup will not notice emulation. A guest whose entire workload is the device will notice everything, and no amount of batching recovers the difference between a trap and no trap.

The scheduling consequences of sharing one device across guests, as opposed to partitioning it, are worked out in Scheduling and Preemption on Accelerator Queues.