Nested Virtualization: Running a Hypervisor Inside a Guest

A hypervisor expects to be the only thing between a guest and the processor. It reads and writes a control structure that the processor consults on every entry and every exit, and it assumes no one else is editing that structure. When a hypervisor runs inside a guest, that assumption fails: the inner hypervisor wants its own control structure, the processor has one per virtual CPU, and the outer hypervisor must make the inner one believe it has exclusive use of a structure that is actually being shared.

The arrangement has three levels with conventional names. L0 is the outermost hypervisor, running in root mode on real hardware. L1 is the hypervisor that runs as a guest of L0, in non-root mode, but still believes it is in root mode with respect to its own guests. L2 is the ordinary guest that L1 runs. L0 must virtualize the hardware for L1, including the hardware features L1 needs in order to virtualize for L2.

The structure that cannot be shared

The control structure is a physical page whose address the processor holds. L0 owns one per virtual CPU and points the processor at it before every entry. L1 also wants one per virtual CPU, addressed by a guest-physical address it controls, and it will read and write that page expecting the processor to honour the contents.

L0’s approach is the same one it uses for page tables, and it has two variants. The simpler is to let L1 write its structure into guest memory, and to compose the two structures in software: before entering L2, L0 reads L1’s fields, merges them with its own, and writes the result into the real structure. On exit, L0 reads the real structure’s exit reason and decides whether it belongs to L1 or to L0, delivering it to L1’s handler or handling it itself.

The composed approach is correct but expensive, because the merge happens on every entry. A structure holds hundreds of fields, and only some of them can be copied wholesale. The alternative, which hardware now supports, is a shadow structure: L0 keeps a second structure that the processor actually uses, and marks L1’s structure’s page read-only so that writes to it trap. The trap handler updates the shadow, exactly as shadow page tables update shadow entries from guest page-table writes.

The two variants have the same trade as their paging equivalents. Composition on every entry costs a fixed amount regardless of how often L1 changes its controls. Shadowing costs a trap per change and nothing otherwise. A hypervisor that sets up its structure once and rarely modifies it prefers shadowing; one that modifies per entry, which is what a hypervisor does when it switches between its own guests, prefers composition or a hardware shadow.

Merging the intercept sets

The interception policy is where nesting becomes semantically interesting. L1 wants certain operations to exit so it can handle them for L2. L0 wants certain operations to exit so it can handle them for L1, or for itself. The rule is a union: an operation exits to L0 if either level asked for it.

That union is necessary but not sufficient. When an operation exits to L0, L0 must decide whether the exit is one L1 expected. If L1 asked for it, L0 must reflect it into L1’s structure, set L1’s exit reason and qualification fields, and enter L1 so its handler runs. If L1 did not ask for it, L0 handles the exit itself, performs whatever emulation is needed, and re-enters L2 without L1 ever knowing.

The classification is not always decidable from the exit cause alone. Some exits are L0’s business regardless, such as an exit caused by L1’s own structure being accessed. Others are L1’s, such as a CPUID exit that L1 wants in order to filter feature bits for L2. A third class is ambiguous: a page fault on a guest-physical address that L0 has mapped but L1 has not, where L1’s own second-level tables need to be consulted before anyone can say which level should handle it.

/* Classify an exit that reached L0 while L2 was running. */
static enum owner exit_owner(struct vcpu *v, uint32_t reason)
{
    if (reason == EXIT_VMX_INSTRUCTION)
        return OWNER_L0;              /* L1 touched privileged structure state */

    if (reason == EXIT_EPT_VIOLATION) {
        /* Did L1 ask to see faults on this region of L2's address space? */
        if (l1_intercepts_gpa(v, v->exit_qualification))
            return OWNER_L1;
        return OWNER_L0;              /* L0's own mapping is what failed */
    }

    return l1_intercepts(v, reason) ? OWNER_L1 : OWNER_L0;
}

The ambiguity in the second branch is the source of most nested-virtualization bugs. L0’s second-level tables map L1’s guest-physical space, which includes the memory L1 uses for L2. L1’s second-level tables map L2’s guest-physical space onto L1’s guest-physical space. A fault on an L2 address can arise in either dimension, and L0 must consult L1’s tables to know which. The two-level walk described in second-level address translation becomes a three-level walk when the guest is itself translating.

The three-dimensional walk

An L2 virtual address is translated by L1’s tables into an L1 guest-physical address, which is translated by L0’s tables into a machine address. Each of those translations is itself a page-table walk whose table reads are addresses that must be translated by the level above.

The reference count grows accordingly. With four-level tables at each dimension, an L2 translation that misses every cache costs up to sixteen memory references for the guest’s own walk, each of which requires up to four references to translate the table address through L1, each of which requires up to four to translate through L0. The nested worst case is therefore on the order of dozens of memory references for a single translation. Hardware caches the intermediates, which is the only reason the arrangement is usable, and the caches are shared across levels so a hit at one dimension helps the others.

The practical consequence is that a nested configuration is disproportionately sensitive to TLB and page-walk-cache pressure. A workload that thrashes the TLB at one level thrashes it at all three, and the miss cost compounds. Implementations mitigate by using large pages wherever the guest’s alignment permits, since a large page collapses a whole level of the walk in whichever dimension it applies.

Exit latency compounds

An exit from L2 to L1 is not one exit. The processor exits to L0, which handles or reflects, and then L0 enters L1 so L1’s handler can run. L1’s handler then does whatever it would have done for L2, which may itself involve an exit to L0 if L1 touches something L0 intercepts. A single logical event can therefore cost two full exit-entry cycles plus whatever the handlers do.

The state that must be saved and restored doubles. L0’s structure holds L1’s state when L1 is running and L2’s state when L2 is running, and the fields that describe L1’s own view of L2 must be preserved across L0’s interventions. Any field L0 modifies to handle an exit must be restored before re-entry, and any field L1 modifies that L0 relies on must be read before it is overwritten. Implementations keep a software copy of the structure’s mutable fields and a generation counter to detect divergence.

There is a class of failure specific to this arrangement: a field that L1 writes but L0 never observes. If L0 enables interception of writes to that field, it sees them. If it does not, L1’s write goes directly to the memory page, and L0’s shadow copy goes stale. The fix is to intercept every field L0 shadows, which is a policy that must be complete: a single missed field produces a guest that works until it exercises the one path that depends on it.

Where nesting is worth its cost

The straightforward use is development and testing: an operating system’s virtualization stack can be exercised inside a virtual machine rather than on dedicated hardware, and a cloud provider can let a customer run their own hypervisor without giving them a physical machine.

The second use is isolation layering. A workload that must not trust the primary hypervisor can run under a second hypervisor that the workload’s owner controls, with the outer hypervisor unable to inspect the inner guest’s memory. That property depends on L0 not being able to read L2’s state, which is not something the architecture guarantees: L0 owns the machine and can read any memory. What nesting gives is not protection from a malicious L0 but a clean administrative boundary, where the inner hypervisor’s owner controls the guest’s configuration and the outer one controls the hardware.

The third use is feature composition. A hypervisor may want to expose a feature to its guests that the underlying hardware does not have, and implement it by running a hypervisor that provides it. That is nesting as emulation, and it costs the full recursion on every operation the emulated feature touches. The alternative is to emulate the feature directly at the outer level, which is cheaper and less general.

The cost accounting is what decides. Nesting doubles the exit path and triples the translation path, and it adds a shadow structure per virtual CPU per level. It is worth it when the alternative is not having the feature at all, and it is not worth it when the alternative is a single-level design with a paravirtualized interface. The rest of this cluster’s hardware-assisted mechanisms, from the control structure to posted interrupts, all compose under nesting, but each one composes by being virtualized again, and the cost of the virtualization is paid by the level that did not ask for it.