Anatomy of a VM Exit: State Save, Exit Qualification, and Where the Cycles Go

A guest instruction that the hypervisor has asked the processor to intercept does not complete in guest context. The processor stops, writes a description of what happened into a region of host memory that the guest cannot address, loads the host’s saved register state, and resumes execution at a host-chosen entry point. That round trip is the VM exit. Everything a virtual machine monitor does for a guest that the guest is not allowed to do for itself is built on top of it, which makes its cost the floor under the cost of emulation.

The interesting engineering is not the switch itself, which the hardware performs in a fixed sequence. It is the information the switch carries, the reasons it fires, and the ratio between the work the processor does and the work the handler does afterwards.

The transition is a memory protocol, not just a privilege change

On Intel VT-x the destination of that write is the virtual machine control structure, a naturally aligned 4 KiB region of host physical memory whose layout is implementation-defined but bounded at 4 KiB. The processor populates a guest-state area with the guest’s general-purpose registers, RIP, RSP, RFLAGS, segment registers with their bases and limits, the control registers, the descriptor table registers, and a set of MSRs selected by the host in advance. It then loads a host-state area that holds the same categories for the host. On AMD SVM the equivalent structure is the virtual machine control block, also 4 KiB, and the save and restore sets are broadly the same.

Two fields matter more than the register image. The exit reason is a 32-bit field whose low 16 bits identify the basic exit cause; the upper bits carry a flag indicating whether the exit came from a nested guest. The exit qualification is a 64-bit field whose bit layout is defined per exit reason. An EPT violation uses it to report whether the access was a read, a write, or an instruction fetch, whether it occurred in guest-physical or guest-linear address space, and whether the faulting access was itself the result of translating a guest page-table entry. A task switch exit reuses the same 64 bits as a segment selector and a task-state-segment descriptor. The handler must therefore know the exit reason before it can parse the qualification, and a handler that guesses will misread memory.

The MSR save and load lists are the part most often overlooked. The processor can be told to store a named set of MSRs into the guest-state area on exit and load another set on entry, which turns the MSR file into part of the context switch. Each list is capped in length, and the ordering is fixed, so a hypervisor that wants a particular MSR preserved must count how many slots it is spending.

None of this is free at the hardware level. The save set is dozens of registers and the load set is dozens more, and both are memory writes and reads to a structure that was recently written by the guest and must not be cached in a way that lets a speculative path observe it. Published comparisons of the two x86 virtualization generations put an exit-entry pair in the high hundreds to low thousands of cycles on server hardware, with wide variance by microarchitecture. That is the constant term. The variable term is everything the handler chooses to do.

Which exits actually fire

A guest that runs a compute loop executes millions of instructions between exits. A guest that runs a network workload may execute a handful of instructions between them. The distribution of exit reasons determines which optimizations matter, and the distribution is not intuitive.

CPUID exits unconditionally. There is no bitmap for it, because the instruction is defined to be intercepted whenever virtualization is active. A guest that calls it in a tight loop, as some runtimes do when probing for features, produces a stream of exits whose handler must synthesize a leaf-by-leaf response from a policy table.

MSR and control-register accesses exit selectively. The MSR bitmap is a 4 KiB structure holding two bits per MSR address, one for reads and one for writes; setting a bit causes an exit, clearing it lets the access execute natively. Reads of the time-stamp counter are usually left native, while writes to the syscall-entry MSRs are intercepted so that the host can validate the target. CR3 writes exit unless the hypervisor enables the guest to load a small set of pre-approved values without leaving, a facility that exists precisely because process switches in the guest are frequent.

MMIO exits because the address is not backed by memory. This is often the dominant exit source for I/O-bound guests and the subject of why emulated device access multiplies.

EPT violations exit because the second-level translation found no mapping or found one without the required permission. These are not errors in the ordinary sense; they are how the hypervisor learns that a guest touched a page it has not yet backed, which is the mechanism behind lazy memory allocation and behind the two-dimensional page walk.

External interrupts exit when the interrupt cannot be delivered directly, and the cost of arranging direct delivery is what posted-interrupt processing exists to avoid.

A rough ranking that holds across workloads: device emulation and interrupts dominate I/O-heavy guests, EPT violations dominate memory-heavy guests during warm-up, and CPUID and MSR traffic dominate guests that probe the platform. A hypervisor that profiles only the total exit count learns little; the reason histogram is the actionable number.

What the handler costs beyond the switch

The processor hands control to the host at a fixed entry point. From there the cost is ordinary kernel code, and it is usually larger than the switch.

The first component is dispatch. The exit reason must be read, the qualification decoded, and control transferred to a handler. On hardware that supports it, a hypervisor can cache the decode in a per-CPU structure so that repeated exits of the same reason skip the full dispatch, but the read of the exit reason field itself cannot be skipped.

The second is the emulation itself. Emulating one instruction fetch requires reading the instruction bytes, which on x86 means decoding a variable-length encoding with prefixes, ModRM, SIB, and displacement. For a MOV to an emulated device register, the handler must determine the operand size, whether the access was aligned, and whether it crossed a page boundary, because a single instruction can produce two exits with different qualifications. Instruction decoders written for this purpose are a recurring source of bugs, and hardware vendors added an instruction-length-decoding facility to the control structure specifically to reduce the work.

The third is re-entry. Entering the guest reloads the host-state fields into the guest-state fields, and the processor must also invalidate translation structures whose validity the exit may have changed. A handler that modifies guest page tables or the second-level tables must request the appropriate invalidation, and a handler that forgets costs correctness rather than time.

The fourth is the accounting. The host must record that time passed, advance the guest’s virtual clock, and check whether the guest has consumed its scheduled slice. On a busy host this check is where a hypervisor discovers it must preempt, which produces a second, larger exit-like event handled by the scheduler rather than the exit machinery.

Measuring the total from inside the guest is possible because some instructions are intercepted unconditionally. Taking the minimum over many iterations filters out scheduler noise, and the residual gap against the same instruction executed without interception is the per-exit cost.

#include <x86intrin.h>

/* CPUID is intercepted with no bitmap to disable it, so the time between
   the counter reads brackets one exit and one entry. The minimum over many
   iterations is the best estimate, since a descheduled vCPU only inflates it. */
static unsigned long long exit_cost_cycles(void)
{
    unsigned long long best = ~0ULL;

    for (int i = 0; i < 1000; i++) {
        unsigned int eax = 0, ebx, ecx, edx = 0;
        unsigned long long t0 = __rdtsc();
        __cpuid_count(0, 0, eax, ebx, ecx, edx);
        unsigned long long dt = __rdtsc() - t0;
        if (dt < best)
            best = dt;
    }
    return best;
}

Making exits rarer instead of faster

Because the fixed cost of the transition resists optimization, the productive strategy is to reduce the count. Every generation of hardware assistance has added a way to let a specific class of operation run without leaving.

MSR bitmaps remove exits for the MSRs a guest is allowed to own. CR masks let the guest read and write certain control-register bits without interception, with the host validating the rest. The APIC-access page lets the guest read and write its local APIC through ordinary memory instructions while the hardware traps only the accesses the host cares about. Posted interrupts let a device interrupt be recorded in guest memory and delivered on the next entry rather than forcing an immediate exit. Nested paging removes the need to shadow page tables and therefore removes the exits that shadowing generated on every guest page-table update, at the cost of making every memory access walk two hierarchies.

The general shape is consistent: the hardware takes over a class of intercepts that software used to handle, and the residual exits become the ones that genuinely require host policy. The performance question then shifts from “how fast is an exit” to “what fraction of guest behaviour still requires one”, which is a property of the workload and the device model rather than of the processor.

The asymmetry that makes exit cost hard to state

A single number for exit latency is misleading because the exit and the entry are not symmetric in what they invalidate. An exit may flush TLB entries tagged for the guest, and an entry may restore them; a hypervisor that keeps the guest’s address space resident pays less than one that does not. The same instruction can cost a few hundred cycles in one configuration and several thousand in another purely because of translation state.

The practical consequence is that exit counts and exit time must be measured together, and that a change which reduces exits can increase total time if it makes each remaining exit more expensive. Device models that batch work, such as a virtio queue processed in one exit rather than one exit per descriptor, trade latency for count, and the trade only pays when the queue is deep. That trade-off is the subject of notification suppression in the virtio ring, where the same reasoning applies to the guest-to-host signal path.