Interrupt Delivery Inside a Virtual Machine

A guest kernel programs its local APIC timer, masks a device line, and takes an interrupt, exactly as a kernel on bare metal does. The difference is that the local APIC it programs may not exist. The physical local APIC belongs to the host, is programmed by the host’s interrupt code, and cannot be handed to a guest without also handing over control of the CPU. Everything a guest does with interrupts therefore either traps to the hypervisor or is emulated by hardware designed to make it not trap.

The cost model is set by which of those two happens. A trap is a world switch: the guest’s register state is saved, the hypervisor runs, the operation is emulated, and the guest is resumed. The switch saves a large register set and flushes part of the guest’s address-translation state, so its cost is hundreds of instructions at a minimum and the dominant term is the time spent refilling caches and translation structures afterwards, not the save itself. Whatever the exact figure on a given processor, the comparison that matters is the one to the interrupt’s own budget: at a timer frequency of a few hundred hertz the cost is negligible, while for a device interrupting tens of thousands of times per second it is the dominant cost of running the workload.

The trap, in detail

Without hardware assistance, the hypervisor intercepts the guest’s writes to the APIC’s memory-mapped page. The page is not mapped to the guest; a write to it causes an exit. The hypervisor decodes the instruction, extracts the register and value, updates its software model of the virtual APIC, and decides whether the write should cause a virtual interrupt to be delivered.

Delivery is the second half. If the guest has not masked the interrupt and its virtual task priority allows it, the hypervisor must make the guest take an interrupt at a point where it is safe. That means either setting a bit in the guest’s virtual interrupt state that the guest’s own entry path will notice, or forcing an exit at the next opportunity and injecting the vector through the virtual machine control structure.

Every timer interrupt in this model is at least one exit. A guest with a 1000 Hz tick and four virtual CPUs generates four thousand exits per second just for timekeeping, before any device work. The exits also perturb the guest’s caches, because the hypervisor’s code and data are different pages than the guest’s.

The emulation has a second cost that is easy to miss. Because the virtual APIC is software, its state must be migrated, saved, and restored with the guest. The state includes the in-service register, the trigger mode configuration, the timer’s initial count, and the current count. A hypervisor that models all of it correctly has a substantial per-VCPU structure, and a bug in the model shows up as a guest that loses interrupts or takes them at the wrong priority.

Letting hardware hold the state

APIC virtualization removes the exit for the common cases by giving the guest’s APIC state a hardware home. The processor holds a virtual APIC page whose layout mirrors the physical one, and the guest’s accesses to its APIC region are redirected to that page without trapping. The guest reads and writes the same offsets it would on bare metal, and the hardware applies the same semantics.

Two pieces of state make this work. The first is the virtual APIC page itself, which is where the guest’s register writes land. The second is a set of controls that tell the processor when a virtual interrupt is deliverable: the guest’s task priority, its interrupt flag, and whether it is in a state where an interrupt can be taken. The processor evaluates those conditions on every virtual interrupt and delivers it directly into the guest without the hypervisor’s involvement.

The effect is that a guest’s timer interrupt becomes a normal interrupt. The physical timer fires, the hardware consults the virtual APIC page, and if the guest is ready the vector is delivered into the guest context. No exit, no emulation, no world switch. The saving is the entire trap cost, and it applies to every interrupt whose destination is a virtual CPU rather than the host.

The catch is that the hypervisor still owns the physical timer. A guest programming its virtual timer is asking for a wakeup at a future time, and the hypervisor must translate that into a physical deadline and arrange for the guest to be running at that moment. That scheduling problem is not solved by APIC virtualization; it is a separate mechanism, and it is why a guest’s timer accuracy depends on how promptly the hypervisor resumes it.

Posted interrupts

The remaining trap is the device interrupt. A physical device’s MSI is delivered to a physical CPU, which is running either the host or some guest. If the destination is a guest, the interrupt must reach that guest’s virtual APIC, and without assistance that means an exit so the hypervisor can decide what to do.

Posted interrupts remove the exit by letting the device’s write be redirected. The interrupt remapping hardware, which already translates the device’s message into a physical destination, is extended to translate it into a posted format: the vector is written to a data structure associated with the target virtual CPU, and a notification is raised to the physical CPU only if the guest is not currently running.

The structure is a descriptor per virtual CPU holding a pending vector bitmap and a notification vector. When a device’s MSI arrives, the remapping unit writes the bit for that vector into the descriptor. If the target VCPU is running, the processor notices the pending bit at the next evaluation point and delivers the interrupt into the guest. If it is not running, a notification interrupt goes to the physical CPU, and the hypervisor wakes the VCPU.

The consequence for a pass-through device is that its interrupt path is close to bare metal: the device writes, the remapping unit redirects, the guest takes the vector. The hypervisor is involved only when the VCPU is not running, which is exactly when it has to be.

Where the costs remain

The first remaining cost is the wakeup. A VCPU that is not running when its interrupt arrives must be scheduled, and the delay is a scheduling decision in the host. For a latency-sensitive guest this is the dominant term, and it is not addressed by any of the interrupt-delivery mechanisms.

The second is the notification itself. Even when the VCPU is running, the notification interrupt to the physical CPU is a real interrupt, taken by the host’s entry path before the guest’s. The posted mechanism reduces the number of exits but does not eliminate the host’s involvement entirely.

The third is the interrupt’s effect on the host’s own scheduling. A guest that generates a high rate of interrupts consumes host CPU time in the notification path, and the host’s scheduler charges time spent running the VCPU to whatever task the VCPU is attached to, so the cost lands on the virtual machine’s own budget rather than on an unaccounted slice of the host. The consequence is that the guest’s view of its own CPU consumption includes delivery work it cannot observe: a guest that looks busy may be paying for interrupts rather than for its own instructions.

The fourth is the quality of the virtual timer. The virtual timer’s accuracy is bounded by the host’s timer and by how promptly the VCPU is resumed. A guest that programs a 100-microsecond deadline may observe a much larger delay if the host does not schedule it in time, and the guest has no way to distinguish a late timer from a lost one.

Emulated devices and the legacy path

Devices that are emulated rather than passed through take the slow path by construction. The emulated device’s interrupt is raised by the hypervisor itself, in software, and injected into the guest. There is no remapping unit involved because there is no physical device.

The model is still worth understanding because it defines the fallback. An emulated network adapter that interrupts once per packet forces an exit per packet, and the cost is the world switch plus the emulation plus the injection. This is why high-rate devices are passed through or handled by a virtio-style interface with batching, where the notification is amortised across many packets and the guest polls.

The batching is the same idea as NAPI, applied at the hypervisor boundary. The device or the hypervisor accumulates completions and notifies once, and the guest’s driver polls. The parameters are a count and a timer, and the trade between latency and exit rate is the same one described in NAPI and Interrupt Mitigation, with the world switch replacing the interrupt entry as the cost being amortised.

What this changes about the guest kernel

Very little, and that is the design goal. A guest kernel uses the same entry path, the same descriptor table, and the same deferral mechanisms as a host. The interrupt it takes is a real interrupt with a real vector, delivered through the same gate.

What differs is the latency distribution. The delays catalogued in Measuring and Bounding Interrupt Latency are all present, plus a term for the hypervisor’s scheduling of the VCPU. That term is not bounded by anything the guest controls, which is why real-time workloads are usually run on isolated physical CPUs rather than in a guest.

The other difference is observability. The guest sees its own interrupt counts and its own handler durations, and it cannot see the exits that its operations caused. A guest that appears to have a latency problem may be measuring the host’s scheduling rather than its own code.

The direction of the mechanism

Each generation of the mechanism has moved work from software to hardware: first the vector assignment, then the delivery, then the state. The pattern is the same as the one in Interrupt Controllers from the 8259A to Message-Signalled Interrupts, where the acknowledge cycle and the shared wire were removed by giving the device the ability to write a message. In both cases the hardware gains a table that the software configures, and the software’s remaining job is to configure it correctly and to handle the cases the hardware cannot.

The cases the hardware cannot handle are the ones that require a decision: which VCPU should receive an interrupt when the device is shared, what to do when the target is not running, and how to keep the timer accurate when the guest is not scheduled. Those decisions are the hypervisor’s, and their cost is the reason the mechanisms exist.