Emulated MMIO: Instruction Decode, Register Semantics, and Exit Storms

A device register is memory that is not memory. The guest writes a 32-bit value to an address, and the effect is not that a location changes but that hardware performs an action: a queue doorbell rings, a descriptor pointer is stored, a status bit is cleared. When the device is emulated in software, the write must not land in RAM. It must stop, be examined, and be translated into a call into a device model. That is what an MMIO exit is for, and the cost is dominated by what the hypervisor has to reconstruct about the instruction that caused it.

From guest store to handler call

The hypervisor carves the guest-physical address space into regions. Regions backed by host memory are mapped through the second-level page tables. Regions that represent device registers are left unmapped, or mapped to a reserved page, so that any access faults. When the guest executes a store to such an address, the processor walks the tables, finds no valid second-level entry, and exits with an EPT violation whose qualification says the access was a write and supplies the guest-physical address.

The handler now knows the address and the direction, and nothing else. It does not know how wide the write was. It does not know the value. Those are properties of the instruction, and the instruction is in guest memory at the guest’s RIP. So the handler reads the instruction bytes and decodes them.

This is the structural weakness of emulated MMIO: the fault reports an address, not an operation. On architectures with a fixed instruction width the decode is cheap. On x86 it is the most complicated decoder in common use, because the encoding is variable-length and the operand size depends on prefixes, the default operand size of the current code segment, and the presence of a REX prefix.

The processor does help. The VMCS contains a field, populated on exit, holding the length of the instruction that caused the exit. The handler can add that length to the saved RIP to know where to resume. It cannot use that field to learn the operand size, so it still has to decode at least the opcode and ModRM bytes to determine whether the store was 1, 2, 4, or 8 bytes and which register supplied the value.

/* Decode enough of a ModRM byte to interpret an emulated device access.
   Returns the guest register index and the access width in bytes.

   The rm field, plus a SIB byte and displacement where present, gives the
   effective address, which the handler already knows from the exit
   qualification. Note that mod == 0 with rm == 5 is RIP-relative. */
static int decode_device_access(unsigned char modrm, int rex_w, int rex_r,
                                int op_size_prefix, int *reg_index)
{
    int mod = (modrm >> 6) & 3;

    /* mod == 3 addresses a register; no memory access, so no exit. */
    if (mod == 3)
        return 0;

    /* The reg field names the register operand, extended to 16 by REX.R. */
    *reg_index = ((modrm >> 3) & 7) | (rex_r << 3);

    /* 0x66 selects 16 bits, REX.W selects 64, otherwise the code segment
       default applies, which is 32 bits for 64-bit and 32-bit segments. */
    if (op_size_prefix)
        return 2;
    if (rex_w)
        return 8;
    return 4;
}

Reading the value without re-executing

Once the operand size is known, the value must be obtained. For a store, the value is in a guest register, and the register is in the saved guest-state area of the control structure, indexed by the register number in the ModRM byte. A store of the low byte of a register requires the handler to select the correct byte from the 64-bit saved value, and the encoding of byte registers depends on whether a REX prefix is present: without REX, the bytes are AL, CL, DL, BL and the high bytes AH, CH, DH, BH; with REX, the byte registers are the low bytes of any of the sixteen registers. A decoder that assumes one mapping produces wrong values for the other.

For a load, the handler computes the value and must place it in the guest register before resuming, which means writing back into the guest-state area and letting the entry sequence reload it.

There is a shortcut that avoids decode entirely for some accesses. If the hypervisor arranges for the access to fault, then maps the faulting page to a host page containing a synthetic value, the guest can complete the load itself. For reads of read-only registers that return a constant, this works and removes the exit. Writes cannot be handled this way, because the write must not be silently swallowed: the device needs to observe it.

What a single doorbell write actually costs

Consider a driver that writes a queue index to a doorbell register. The path is: guest store, EPT violation, exit to host, save of the register file, dispatch on exit reason, decode of the instruction, extraction of the register value, lookup of the device model, call into the model, model updates internal state, model may signal a worker thread or kick a host-side processing loop, return, request TLB invalidation if the mapping changed, entry to guest. The guest executed one instruction.

The number of cycles is dominated by the handler, not the transition, and the handler’s cost is dominated by cache misses: the instruction bytes, the device model’s state, and the control structure are three separate working sets that a busy host will not keep hot. A device model that must take a lock adds contention across vCPUs, and a model that must wake a thread adds a scheduler interaction.

The storm case arises when a driver polls. A driver waiting for a device to become ready may read a status register in a tight loop. Each read is an exit. A loop that iterates a few thousand times per operation, across many virtual machines, produces a host that spends its time in the exit handler rather than in guest code. The guest’s own time accounting shows the guest as busy, because from the guest’s perspective the instructions executed; the cost appears as host system time.

Why batching changes the shape

The mitigation is to make one exit accomplish more work. A virtio queue is the canonical example: the driver writes descriptors into a ring in ordinary guest memory, which needs no exit at all, and then rings a doorbell. One MMIO write signals that an arbitrary number of descriptors are available. The hypervisor processes the entire batch in one handler invocation.

The economics are clear. If a request costs one exit plus per-descriptor work, and the per-descriptor work is small, then the exit is the fixed cost and batch size is the lever. Deep queues amortize; shallow queues do not. A workload that submits one request at a time and waits for completion cannot batch, and therefore pays the full exit cost per request no matter how efficient the device model is.

The same logic explains notification suppression in the virtio ring: the guest can tell the device not to interrupt it, and the device can tell the guest not to notify it, both to reduce the number of boundary crossings in either direction.

Register semantics that emulation gets wrong

An emulated device must reproduce not just the values of its registers but their side effects and their read-back behaviour. Real hardware often has registers where a read clears a bit, where a write of 1 clears rather than sets, where a write of 0 has no effect, or where a register returns different values on consecutive reads. A model that treats every register as a plain storage cell will appear to work for drivers that only write and then poll for a documented value, and will fail for drivers that rely on write-one-to-clear semantics.

Width matters too. A driver may perform a 32-bit write to a register that the model defines as 8 bits, expecting the upper bits to be ignored, or may perform two 16-bit writes to compose a 32-bit value. A model that rejects the access, or that applies it as a full-width store, diverges from hardware. The specification for the device usually states the required widths, and where it does not, the model must match the behaviour of the hardware the guest was written for.

There is a further trap in the ordering of accesses. A driver that writes a command register and then reads a status register depends on the two accesses arriving in order, which they do, because each is a separate exit handled on the same vCPU. But a driver that writes a command and then expects a device-initiated memory write to be visible depends on memory ordering between the emulated device’s DMA and the guest’s reads. The hypervisor must ensure that the DMA write is visible before the status bit that signals its completion, which is the same ordering discipline a real device needs, implemented with a memory barrier rather than with bus protocol.

The alternative: give the guest a real device

The exit storm disappears if the guest talks to hardware directly. Passthrough assigns a physical device to the guest through an IOMMU, so that the guest’s MMIO accesses are decoded by the device rather than by software, and the guest’s DMA addresses are translated by the IOMMU rather than by the hypervisor. The guest driver runs unmodified at native speed.

The cost is granularity. A passed-through device belongs to one guest; it cannot be shared without a hardware mechanism for partitioning, and the host loses the ability to inspect or emulate it. Migration becomes difficult because device state lives in hardware that may not exist on the destination. Interrupt delivery must be routed to the guest, which is the problem posted interrupts address.

Between the two extremes sits the paravirtual device, which is emulated but designed for emulation: a small number of registers, accesses batched through shared memory rings, and a specification that both sides implement. The performance ceiling is lower than passthrough and the exit count is far lower than a model of real hardware, which is why it became the default for I/O in virtual machines.

Measuring the right thing

Counting exits per second is not enough to diagnose an MMIO problem, because the cost per exit varies by an order of magnitude depending on whether the handler hits in cache and whether the device model contends. What distinguishes a healthy configuration from a pathological one is the ratio of exits to useful guest work, and the reason distribution within the exits. A guest with a high rate of EPT violations on device pages and a low rate of violations on data pages is spending its budget on register pokes rather than on computation, and no amount of handler optimization will fix a driver that polls.