Inside the VMCS: What Hardware Virtualization Actually Saves and Restores

Ring deprivileging made the guest kernel a second-class citizen of the processor. Hardware virtualization makes it a first-class citizen of a smaller processor. The guest runs at ring 0 with the full instruction set available, and the processor switches to a separate operating mode whenever the guest does something the hypervisor has declared interesting. The switch is not a fault the guest can observe; it is a mode change that the hypervisor handles and then reverses.

The mechanism is a memory structure the processor reads and writes. Intel calls it the virtual machine control structure, AMD’s equivalent is the virtual machine control block, and both serve the same purpose: they hold the guest’s saved processor state, the host’s saved processor state, and a set of controls that determine which guest actions cause a switch. Everything about the design follows from what that structure can express.

Root mode, guest mode, and a structure in between

The processor gains a new dimension of privilege. In VMX operation, code runs either in root mode, which is where the hypervisor lives, or in non-root mode, which is where the guest lives. Non-root mode has its own rings 0 through 3, so the guest kernel gets ring 0 and the guest’s user processes get ring 3, exactly as on bare metal. A transition from non-root to root is a VM exit; the reverse is a VM entry.

The control structure is per-virtual-CPU, not per-machine, and it is referenced by a physical address held in a processor register that the hypervisor sets before entry. The structure is not in the guest’s address space and the guest cannot name it. It contains four groups of fields.

Guest-state fields hold the processor state the guest expects to resume with: control registers, segment registers with their cached descriptors, the descriptor table bases and limits, the flags register, the instruction pointer, the stack pointer, the page-table root, and the model-specific registers that affect guest execution. Host-state fields hold the same set for the hypervisor, loaded on exit. Control fields decide behaviour. Read-only fields report why the last exit happened.

The exit information field is the important one for the handler. It carries a number identifying the cause, a qualification field whose meaning depends on that cause, and a guest-linear-address field for exits that faulted on a memory access. A handler dispatches on the cause number, uses the qualification to decide what to do, and never has to decode the instruction stream to find out why it was entered.

Execution controls decide the trap set

The execution controls are the policy. They are a bitmap in which each bit enables interception of a class of operation. A guest CPUID exits only if the corresponding bit is set. A guest write to a control register exits only if that register’s bit is set. A guest access to a port exits only if the I/O bitmap marks that port, and the bitmap is a separate region whose granularity is one bit per 16-bit port, so a guest with a large port range costs 8 KiB of bitmap.

Two controls deserve separate treatment because they replace whole software subsystems.

The MSR bitmap does for model-specific registers what the I/O bitmap does for ports. A guest RDMSR or WRMSR exits only for the registers the bitmap marks. This matters because a modern processor exposes hundreds of model-specific registers, most of which are performance counters or feature flags that a guest can read harmlessly, and intercepting all of them would make every CPUID-adjacent code path in the guest expensive. The hypervisor marks the small set that must be virtualized, typically the system call target registers, the APIC base, and the registers that control features the guest must not enable.

The exception bitmap decides which guest exceptions cause exits. An exception the hypervisor does not care about is delivered to the guest’s own handler without leaving non-root mode, which is what makes guest page faults cheap: the guest’s page fault handler runs at native speed, and the hypervisor never sees the fault. The hypervisor sets the bit only for exceptions it must observe, such as those raised by its own shadow structures or by the second-level translation it manages.

The primary processor-based controls also include the interrupt-window and NMI-window exiting bits. These are the mechanism by which a hypervisor that has temporarily blocked interrupts in the guest arranges to be told when the guest is ready to receive them again. Without them, a hypervisor holding a pending virtual interrupt would have to poll the guest’s interrupt flag, which means either exiting on every flag change or running a timer that wakes it unnecessarily.

/* Configure a vCPU so that the interesting operations exit and the rest run. */
static void vcpu_setup_controls(struct vcpu *v)
{
    uint32_t exec = 0;

    exec |= EXEC_CPUID;              /* feature bits must be filtered */
    exec |= EXEC_HLT;                /* the guest may not halt the machine */
    exec |= EXEC_INVLPG;             /* invalidation must reach the shadow layer */
    exec |= EXEC_MOV_DR;             /* debug registers are host state */
    exec |= EXEC_CR3_LOAD | EXEC_CR3_STORE;   /* address space switches */
    exec |= EXEC_IO;                 /* port access goes through the bitmap */

    v->exec_controls = exec;
    /* The exception bitmap stays clear: guest faults are the guest's business. */
}

What an exit actually costs

An exit is not a trap. A trap saves a few registers and vectors through a handler the guest installed. An exit saves the guest’s architectural state into the control structure, loads the host’s state, switches to root mode, and enters the hypervisor at an address the structure specifies. An entry reverses all of it. The processor does this in hardware, but the state saved is far larger than a trap frame, and the memory traffic is the cost.

Two further costs are less obvious. The first is that the hypervisor’s exit handler runs with the guest’s TLB entries still present, tagged by whatever address-space identifier scheme is in use, so an exit that changes mappings must invalidate carefully. The second is that the guest’s caches hold data the hypervisor has just modified in the control structure, and the structure’s cache line is shared between the processor’s exit path and the hypervisor’s handler.

The magnitude matters because exits are counted per instruction. A guest executing a tight loop of port accesses generates one exit per access, and each exit costs hundreds to thousands of cycles depending on what the handler does. A guest that never exits runs at native speed. The design goal is therefore always the same: keep the exit count proportional to operations the guest performs rarely, and push anything frequent into a path that does not exit.

The gap between a trap and an exit

The most consequential difference between hardware assistance and trap-and-emulate is what happens to guest state that the hypervisor wants to change without the guest noticing. A trap gives the hypervisor the guest’s registers on a stack and lets it modify them. An exit gives the hypervisor a control structure, and the fields in that structure are not all equivalent.

Some fields are loaded on entry and therefore fully under hypervisor control: the guest’s control registers, its segment descriptors, its page-table root. The hypervisor can change any of them before the next entry, and the guest will resume with the new value. Others are written by the processor during guest execution and read by the hypervisor on exit, such as the exit qualification and the guest-linear address. The hypervisor reads those, but changing them changes nothing.

The asymmetry produces a class of bug that software virtualization does not have. A hypervisor that wants to hide a control-register bit sets the guest-state field, and the guest reads back the value the hypervisor wrote. But if the guest writes the register while the corresponding intercept is disabled, the processor updates the guest-state field directly and the hypervisor never learns. Re-enabling the intercept afterwards does not retroactively reveal the change. Hypervisors therefore keep a shadow copy of every field they care about and enable interception for the writes that could diverge, which reintroduces exactly the trap-per-write cost that the hardware was supposed to remove.

The general rule that emerges is that hardware assistance moves the cost from “every sensitive instruction” to “every sensitive instruction the hypervisor chose to intercept”, and the choice is constrained by how much state the hypervisor can shadow. Where the state is small, as with control registers, the hypervisor intercepts writes and keeps a copy. Where the state is large, as with page tables, the hypervisor uses the hardware’s own second-level mechanism and stops shadowing entirely.

Nested virtualization and the recursion

A hypervisor running inside a guest must itself use the control structure, but the processor has one. The L0 hypervisor therefore virtualizes the structure for the L1 hypervisor, which means intercepting L1’s accesses to its own structure, translating the guest-physical address of that structure into a machine address, and reflecting L1’s controls into L0’s. Every exit L1 expects must be synthesized by L0, and every exit L1 does not expect must be hidden.

The cost is not one extra exit per event but two, because L1’s handler runs in non-root mode under L0 and its own operations may exit again. The state that must be kept consistent grows to include two copies of every control, and the merging rule is that an operation exits if either level asked to intercept it. Nested virtualization is thus a composition of two virtual machines where the inner one believes it has the hardware, and the outer one pays for the belief.

The design constraints that made hardware assistance attractive in the first place still hold at each level: the guest runs unmodified, the intercept set is a policy, and the cost is proportional to the policy’s breadth. What changes is that the policy is now written by two independent parties with no shared view of the exit budget.