Virtualizing the Local APIC: Access Pages, Virtual Interrupt Delivery, and Posted Interrupts

An interrupt is the one event that cannot be deferred to a convenient moment. The guest’s timer fires, a device completes a request, another vCPU sends a reschedule IPI, and the guest’s interrupt controller must record the request, decide whether it is masked, and deliver it to a processor that is currently executing guest code. The local APIC is memory-mapped: the guest reads and writes registers through ordinary loads and stores at a fixed physical address. That single design decision makes interrupt virtualization a memory-access problem as much as an interrupt problem.

The APIC page and the access trap

The local APIC’s registers live in a 4 KiB page, conventionally at a guest-physical address near the top of the 4 GiB range, though the address is set by the guest’s firmware and the hypervisor must model whatever the guest expects. The registers include an identifier, a version, a task priority register, an end-of-interrupt register, a spurious-interrupt vector register, a timer with its initial count and current count, and the interrupt command register used to send IPIs, plus the in-service and trigger-mode registers that record delivery state.

In the simplest model, that page is not backed by host memory, so every access to it faults. The guest writes the end-of-interrupt register after handling each interrupt, and each write becomes an exit, produced by the same second-level fault machinery that emulated device registers rely on. A guest servicing a high rate of interrupts therefore generates at least one exit per interrupt before any device work is considered. This is a structural cost, not a pathological one, and it was the dominant interrupt overhead on first-generation hardware assistance.

The first mitigation is to back the page with real host memory containing a synthetic APIC image and trap only the accesses that need host involvement. Reads of the identifier and version can be answered by the memory image. Writes to the end-of-interrupt register change state that the host needs to know about, but they can be recorded in the image and examined lazily.

The second mitigation is a hardware feature: a control flag that makes the processor treat a designated guest-physical page as the APIC-access page. Accesses to that page do not fault; instead the processor virtualizes them, which for a write means it may perform the write to the synthetic image without exiting and only exit for the specific registers that require emulation. This removes the exit from the common path entirely, and it is why the APIC page must be 4 KiB aligned and must not be otherwise mapped.

Delivering an interrupt without an exit

Recording a pending interrupt is only half the problem. It must arrive at the guest’s instruction boundary with the right vector and the right priority, which means the host must arrange for the processor to enter the guest with an event already queued.

The control structure has a field for exactly this: an entry-interruption-information field whose valid bit, vector, and type tell the processor to inject an event as part of the entry sequence. The hypervisor sets it before entering the guest, and the guest sees the interrupt as if it had arrived during execution. The guest cannot distinguish an injected interrupt from a hardware one, which is the point.

The cost is that the injection is decided at entry time. If the guest is already running, a newly arrived interrupt cannot be injected until the next exit, because the hypervisor is not executing. The processor can interrupt a running guest, but doing so causes an exit, and the exit is what the hypervisor wanted to avoid. For a guest running long stretches of code, this means interrupts wait for the next natural exit, which can be arbitrarily far away. A guest computing in a tight loop with no device activity might not exit for a long time, and a timer interrupt that should have fired has to wait.

The posted interrupt descriptor

The hardware answer is to let a device or a sending processor write the interrupt into memory that the guest’s own processor will read, with no host involvement at all. This is posted interrupt processing, and it works through a per-vCPU structure called the posted interrupt descriptor.

The descriptor contains three fields that matter. A notification vector names an interrupt vector that the host has reserved for this purpose. A notification destination identifies the physical APIC and delivery mode to use when the host needs to wake a blocked vCPU. A pending bitmap is 256 bits, one per vector, recording which interrupts have been posted but not yet delivered. Alongside it, an outstanding notification field records whether a notification is currently in flight, so that a second event can be coalesced rather than generating a second interrupt.

The flow for a device interrupt is this. The device performs an interrupt remapping table lookup, which translates the device’s identifier and vector into a destination. If the destination is a vCPU with posted interrupts enabled, the interrupt controller does not deliver to the physical APIC. Instead it atomically sets the corresponding bit in the pending bitmap and, if the vCPU is not already running with a notification outstanding, sends the notification vector to the vCPU’s physical APIC. The processor receives the notification, and because the vector is designated as a posted-interrupt notification in the control structure, it does not exit. It performs the posted-interrupt processing in hardware: it reads the pending bitmap, merges the bits into the virtual interrupt state, clears them, and continues executing the guest. If the guest has interrupts enabled, the highest-priority pending vector is delivered immediately. If not, the bit stays in the virtual pending state and is delivered when the guest unmasks.

The result is that a device interrupt delivered to a running guest costs no exit. That is the whole purpose, and it converts a per-interrupt exit into a per-interrupt memory write plus a hardware-assisted merge.

Hardware decides deliverability from the vector’s priority class, which is the high nibble of the vector, compared against the processor priority, which is the higher of the task priority register and the class of the highest vector currently in service. A guest that raises its task priority to mask interrupts below a threshold therefore keeps posted vectors pending without any host involvement, and the merge happens on the next hardware-assisted notification.

/* x86 APIC priority rules in outline: the top four bits of a vector are its
   priority class, and an interrupt is deliverable only when its class is
   strictly greater than the processor priority. */
static int is_deliverable(unsigned char vector, unsigned char tpr,
                          unsigned char in_service_max)
{
    unsigned int tpr_class = tpr >> 4;
    unsigned int isr_class = in_service_max >> 4;
    unsigned int processor_priority = tpr_class > isr_class ? tpr_class : isr_class;

    return (vector >> 4) > processor_priority;
}

The blocked-vCPU case and the wake-up path

A vCPU that is halted is not executing, so there is nothing to merge into. The pending bitmap accumulates bits, and the notification destination field is used: the host sends a wake-up interrupt to the physical processor, which brings the vCPU out of the halted state so the hypervisor can enter it and let hardware deliver the pending vectors.

This is where the design shows its seams. The notification destination is a physical APIC identifier stored in the descriptor, and it must be updated whenever the vCPU migrates to a different physical CPU. If the descriptor is stale, the wake-up goes to the wrong processor and the vCPU stays blocked while the interrupt sits pending. The update must be ordered against the device’s use of the descriptor, which is why the interrupt remapping table entry points at the descriptor and why the sequence for changing a vCPU’s affinity involves disabling posted interrupts, draining outstanding notifications, updating the descriptor, and re-enabling.

A second seam is the interaction with the guest’s own IPIs. A guest sending an IPI to another vCPU writes the interrupt command register, which may itself be an emulated access. The hypervisor can translate that into a posted interrupt to the target vCPU, but only if the target’s descriptor is valid and the vector is in the guest’s range. A guest that sends many IPIs, as a kernel does for TLB shootdowns and scheduler wake-ups, exercises this path constantly.

Nested and non-root complications

When the hypervisor itself runs inside a virtual machine, the posted interrupt descriptor belongs to the outer hypervisor, and the inner one must not write to it directly. Hardware provides a shadow for the virtual APIC state so that the inner hypervisor can enable posted interrupts for its own guests while the outer one still controls the real descriptor. The performance benefit survives, but the configuration becomes a two-level problem: the inner hypervisor’s notification vector must be one the outer hypervisor permits, and the inner one must not assume that a notification it did not send will arrive.

This matters for nested paging as well, because the descriptor and the pending bitmap are host physical addresses, and in a nested configuration the guest-physical address of a descriptor is not the address the hardware must use.

What the alternative costs

Without posted interrupts, an interrupt to a running guest requires an exit. The exit interrupts whatever the guest was doing, which is correct but expensive: the handler must determine the source, decide whether to inject immediately or defer, and re-enter. For a guest handling a hundred thousand interrupts per second, which a busy network workload can reach, the exit cost alone dominates the useful work.

The alternative of injecting at entry only, without exits, avoids the exit but introduces latency: the interrupt waits for the next exit, and a guest that exits rarely experiences interrupt latency measured in milliseconds rather than microseconds. For a timer this is tolerable. For a device completion it is not, because the guest’s request rate is limited by how quickly it learns of completions.

The design that results is a hybrid: posted interrupts for the case where the vCPU is running and the descriptor is valid, entry injection for the case where it is not, and full exits for the cases hardware cannot handle, such as a guest accessing a register the synthetic image cannot emulate. Understanding which of the three paths a given interrupt takes determines whether a guest’s throughput keeps rising with load or flattens once the interrupt path saturates.