Virtual Interrupts: APIC Emulation, Interrupt Remapping, and Posted Delivery

An interrupt is the one event a processor cannot defer. It arrives asynchronously, it preempts whatever is executing, and it carries a vector number that selects a handler from a table the operating system installed. On bare metal the operating system owns the interrupt controller, programs the vector table, and receives every interrupt directly. In a virtual machine none of those three can be true without mediation, because the controller is a single machine-wide device and the vector table is the processor’s.

The problem has two halves that are often conflated. One is the local controller inside each processor, which handles the timer, the inter-processor interrupts, and the delivery of everything the I/O controller forwards. The other is the I/O controller, which collects device interrupts and routes them to a destination processor. A guest must be given a convincing illusion of both, and the mechanisms differ enough that they were solved separately.

Emulating a controller the guest can program

The local interrupt controller is memory-mapped. A guest writes to a page of physical addresses to set up its timer, to send an inter-processor interrupt, to acknowledge an interrupt, and to set the task priority that masks lower-priority vectors. The page is at a fixed physical address, so the guest’s accesses to it land in a region the hypervisor can intercept by making those pages absent from the second-level tables.

Intercepting every access gives the hypervisor a complete model: it holds a software copy of the controller state, applies the guest’s writes to the copy, and injects the resulting interrupts into the guest through whatever mechanism the processor provides. The cost is an exit per access, and the access rate is not low. A guest that acknowledges interrupts, writes the end-of-interrupt register, and reprograms the timer does several accesses per interrupt, so the model charges several exits per interrupt delivered. That is the worst possible place to spend exits, because the work being done is trivial.

The alternative is to give the guest direct access to a hardware controller that the processor can partition. Modern processors can present multiple interrupt controller interfaces, each associated with a virtual machine, and the guest’s writes to the memory-mapped page go to its own interface without an exit. The processor delivers the interrupt to the guest’s virtual processor with the guest’s vector, using a table the hypervisor configures.

Direct access has a constraint the emulated model does not: the guest’s writes must not be able to affect another guest. The partition is enforced by the processor, and the hypervisor’s job shrinks to configuring which interface belongs to which virtual CPU and translating the guest’s physical address for the page into the right interface. Where the hardware cannot partition, the emulated model remains, and the exit cost remains with it.

Interrupt remapping before the interrupt reaches a guest

A device assigned to a guest raises interrupts by writing a message to a physical address, the way it always has. The message contains the destination processor, the vector, and a delivery mode. If the hypervisor let that message through unchanged, the device could target any processor, deliver any vector, and the guest would be able to influence the host by programming its device’s interrupt descriptor.

The IOMMU sits between the device and memory and intercepts these writes. Its interrupt remapping tables translate the message’s address and data fields into a canonical form: a source identifier derived from the device’s requester ID, and an interrupt index that selects an entry in a table the hypervisor owns. The entry names the destination and vector that the hypervisor permits, which may differ entirely from what the device wrote.

The remapping does two jobs at once. It confines each device to the interrupts the hypervisor has allocated to it, which is the isolation requirement. And it makes the interrupt composable: a remapped interrupt carries an identity that says which guest and which virtual CPU it belongs to, so the hypervisor can deliver it to the right place without guessing from the address.

The tables are hierarchical in the usual way. A root table indexed by the requester ID’s bus, device, and function numbers points to a context entry, which points to a remapping table, which is indexed by the interrupt index from the message. The depth is three levels of memory reference per interrupt, cached by an IOMMU-side cache, and the cost of a miss is comparable to a page-walk miss. That cost is why interrupt remapping is often enabled only for devices that are actually assigned to guests, and why the entries are allocated sparsely.

Injection, and the exit it hides

Once the hypervisor knows an interrupt is destined for a guest, it must make the guest’s processor take it. The naive method is to record a pending interrupt in the virtual CPU’s state and, on the next entry, ask the processor to deliver it. If the guest is running, the hypervisor must force an exit to set the pending bit, which is the classic interrupt-window problem: the hypervisor has something to deliver but no way to get the guest’s attention without interrupting it.

The processor’s answer is a window. An execution control lets the hypervisor request an exit when the guest becomes ready to receive interrupts, meaning when its interrupt flag is set and no higher-priority interrupt is being serviced. The hypervisor enables the window, and the processor exits as soon as the guest clears the condition that blocked delivery. The window is closed again on delivery, so the exit happens once per blocked period rather than once per check.

The better answer removes the exit entirely. A posted interrupt is a data structure in memory describing an interrupt that should be delivered to a virtual CPU. When the interrupt arrives, hardware writes the vector into the structure, marks it pending, and, if the target virtual CPU is currently running in guest mode, delivers it directly without leaving non-root mode. The hypervisor is notified only when the target is not running, in which case it will deliver the interrupt on the next entry anyway.

/* Posting an interrupt: no exit if the target vCPU is running in guest mode. */
static void post_interrupt(struct vcpu *target, uint8_t vector)
{
    struct posted_desc *pd = target->posted_desc;

    /* The descriptor is in host memory, not guest memory, and not guest-writable. */
    pd->vector = vector;
    __sync_synchronize();             /* descriptor before the notification */

    if (target->running_in_guest) {
        /* A write to the notification address makes the processor deliver it. */
        write_pir_notification(target);
    } else {
        target->pending_vectors |= (1u << vector);
    }
}

The gain is largest exactly where virtual machines spend most of their time: a network or storage device completing requests while the guest is executing. Each completion becomes a memory write instead of an exit plus a re-entry, and the guest’s interrupt handler runs without the hypervisor being involved at all.

The ordering and coalescing problems that remain

Removing the exit removes the hypervisor’s chance to serialize. Two interrupts posted to the same virtual CPU can be delivered in either order, and a guest driver that assumes a completion arrives after the request it acknowledges is relying on an ordering the hardware does not guarantee. The specification defines the ordering rules, and they are weaker than a naive driver expects: interrupts of the same priority may be delivered in any order.

Coalescing is the second problem. A device that generates an interrupt per packet will post an interrupt per packet, and the guest’s handler will run once per packet even though it could have processed all of them in one visit. The standard remedy is to disable the device’s interrupt and let the guest poll, which is a policy the driver chooses. Hardware cannot coalesce on the guest’s behalf, because it does not know how long the guest is willing to wait.

A third problem is the interaction with the guest’s own masking. A posted interrupt that arrives while the guest has masked the vector must be held, not dropped. The hardware records it as pending, and delivery is deferred until the guest unmasks. If the hypervisor also holds a pending bit for the same vector, the two can disagree about whether the interrupt has been delivered, and a guest that masks, unmasks, and expects exactly one interrupt may receive two or none.

What the guest can still observe

A guest that reads its controller’s state sees what the hypervisor or the partitioned hardware presents. Three leaks recur.

The identifier of the local controller is architectural and usually fixed, so the guest cannot tell from it that it is virtualized. The timer, however, is driven by a clock the hypervisor controls, and a guest that measures the interval between two timer interrupts can detect that the hypervisor descheduled it. Paravirtualized guests read time through an interface that accounts for this, as described in the hypercall ABI.

The set of vectors available is smaller in a virtual machine than on bare metal, because the hypervisor reserves some for its own use. A guest that programs a vector the hypervisor has reserved receives no interrupt and does not know why. Hypervisors therefore either hide the reserved range from the guest’s controller model or remap it, and the remapping is invisible until the guest inspects a register that reports the range.

The destination of an inter-processor interrupt is another leak. A guest with more virtual CPUs than the host has physical ones cannot deliver an interrupt to a specific physical processor, because the mapping changes as the hypervisor schedules. The guest’s request is honored in the sense that some virtual CPU receives it, but the physical destination is not the one the guest named, and the guest’s assumption that a named processor will handle the work is wrong. Multi-processor guests that rely on directed interrupts for cache locality behave differently under virtualization for exactly this reason.

The division of labour

Three mechanisms now share the work. The IOMMU decides whether an interrupt is permitted and which guest it belongs to. The processor’s posted-interrupt path delivers it without an exit when the target is running. The hypervisor’s emulated controller handles the cases the hardware cannot, which are the ones where the guest’s own configuration is too unusual for the partitioned interface.

The design constraint that produced this split is the same one that produced second-level translation: the hypervisor’s per-event cost must be removed from the paths the guest uses constantly. Interrupt delivery is the most constant of all, so it received the most hardware. What remains in software is configuration and the edge cases, which is the same residue that every hardware-assisted mechanism leaves behind.