Hardware Dirty-Bit Tracking for Guest Memory

Migration needs to know which guest pages changed since the last pass. The naive answer is to trap every guest store, which is unusable: a guest writing a few hundred megabytes per second would generate hundreds of thousands of exits per second, each costing on the order of a microsecond or more, and the guest would slow to a crawl.

The answer that works is to let the CPU’s second-level address translation record the write, using the same page-table entries that already map guest physical memory to host memory. The dirty bit lives in the nested page table, so marking a page dirty costs the guest nothing at all.

Where the bit lives

With hardware virtualization, the guest’s own page tables translate virtual addresses to guest physical addresses. A second set of tables, walked by the CPU after the first, translates guest physical to machine addresses. Intel calls these extended page tables (EPT); AMD calls the equivalent nested page tables (NPT); Arm calls it stage 2 translation.

Each leaf entry in those tables has permission and status bits. On Intel EPT, the entry carries read, write, and execute permission bits alongside an accessed bit and a dirty bit; the CPU sets the dirty bit when a write is permitted and actually performed through that entry, and the hypervisor may clear it at any time. On Arm, the stage-2 descriptor has the S2AP permission field and a dirty-bit-modifier that lets a clean, writable entry become dirty on first write rather than trapping. AMD’s NPT entries historically carried no dirty bit, which forces software write-protection on those parts.

When the hypervisor wants a clean slate for a pass, it walks the guest’s nested page tables and clears the dirty bit on every leaf entry, leaving the entries writable. The guest keeps executing at full speed. The first store to a page after the clear causes the CPU to set the bit in place, with no exit and no hypervisor involvement. Reading the bit back means walking the same tables again and collecting the set of dirty leaves.

The cost profile is therefore: one table walk per pass to clear, one per pass to collect, and zero per guest write. That is the whole reason this mechanism exists.

The first pass and CPUs without a dirty bit

A CPU that can set the dirty bit in place is not the only case the hypervisor has to handle, and the fallback explains a design detail that otherwise looks arbitrary. If the hardware cannot record writes, the hypervisor must instead clear the write permission on the nested entries and take a fault on the first write to each page. The fault handler marks the page dirty in a software bitmap and re-enables write permission for that page. That is one exit per page per pass, not one per write, which is tolerable: a guest with a 4 GiB image touches at most about a million pages, and a typical pass touches far fewer.

The first pass of a pre-copy migration uses exactly this write-protect mechanism even when hardware dirty bits are available, for a specific reason: the initial pass copies the entire image, and the bitmap has to describe writes that happened during that copy. Clearing dirty bits before the pass would do nothing, because the bits are already clear on a fresh guest. Write-protecting the entries makes the first write after the copy of each page trap, which is what populates the bitmap for the second pass.

On subsequent passes, where the entries are already known-writable and the hardware supports the bit, the hypervisor clears dirty bits instead of write permissions, and the exits disappear.

Reading the bitmap without losing writes

Collecting the dirty set is not a simple read. The guest is running, so a page can be written between the moment the hypervisor reads its entry and the moment it finishes walking the tables. A page written after its entry was read appears clean in this pass and would be dropped.

Implementations handle this by ordering the collection so that the risk window is bounded and the loss is impossible. The standard sequence clears the bits for the next pass before or while collecting the current pass’s set, so a write that races the collection is attributed to the next pass rather than lost. Concretely:

1. Read and record the current dirty leaves.
2. Clear those same leaves’ dirty bits.
3. Send the recorded pages.
4. Repeat.

A write that lands between step 1 and step 2 leaves the bit set after the clear only if the clear loses the race; a write that lands after step 2 is recorded for the next pass. Either way no write is dropped, at the cost of some pages being reported twice. Duplicate reports are harmless — the page is simply sent again — whereas a dropped write corrupts the guest.

There is a subtlety with huge pages. If the nested entry maps 2 MiB, the dirty bit covers the whole 2 MiB, so a single store marks 2 MiB dirty. The hypervisor can split the entry into 4 KiB leaves when it needs finer tracking, but splitting costs a TLB flush and a table walk on the next access, and the split tables consume memory. Splitting during migration is common for exactly this reason, and it is why a migrating guest’s TLB behaviour changes measurably even though its instruction stream does not.

The bitmap is per-vCPU work too

Nested page tables are shared by all vCPUs of a guest, but the act of clearing and collecting them is not free of concurrency. Multiple vCPUs can be writing different pages simultaneously, and each write sets its own bit; that part is safe because the hardware updates bits independently.

The problem is the clear. If one thread clears dirty bits while another collects them, the collector can miss a page that was written between the clear and the collect, or report a page that was cleared just before being written. Hypervisors serialize the two operations against each other, which means the collection thread must either stop the vCPUs briefly or coordinate through a lock that the vCPU exit paths also take. The usual compromise is to stop the vCPUs for the clear-and-collect pair, which is short because it touches only page-table memory, not guest memory.

This is one of the places where the freeze cost creeps back in: not as data transfer, but as a serialization point whose duration scales with the number of nested page-table entries. For a guest with a large, fragmented memory mapping, walking millions of entries takes longer than walking thousands.

Limits the mechanism imposes on the guest

Because dirty tracking is implemented through the nested page tables, it inherits their constraints:

• Page granularity is the tracking granularity. There is no sub-page dirty information. The amplification discussed in Dirty Page Rate and the Convergence Condition is a direct consequence.
• Memory the guest maps with unusual attributes can be awkward. Device memory and write-combining mappings may not have dirty bits that behave like RAM; the hypervisor generally treats them as always dirty or as non-migratable.
• Assigned devices bypass the tables. A device doing DMA through an IOMMU writes guest memory without going through the CPU’s nested page tables, so no dirty bit is set. Either the IOMMU’s own dirty tracking is used (where available), or the device is unplugged before migration. This is why passthrough devices and live migration have historically been incompatible.
• Nested virtualization compounds the cost. A guest that runs its own hypervisor has two levels of translation, and the outer hypervisor’s dirty tracking then covers memory that the inner hypervisor’s guest writes. The bits still work, but every write is translated twice and the table walk during collection is longer.

The mechanism is also not free in memory: the nested page tables themselves must exist for every mapped region, typically a few bytes per 4 KiB page plus the upper levels, and the migration stream does not carry them because they are host state, not guest state. The destination rebuilds them from the guest’s memory map, which is fast but does mean the destination’s tables can be laid out differently from the source’s — with different huge-page coverage and therefore different future dirty-tracking behaviour.

What the tracking does not tell you

The dirty set says which pages changed; it says nothing about ordering or about how many times a page changed. A page written once and a page written a million times are indistinguishable in the bitmap, so the estimator in the convergence loop cannot tell a hot page from a cold one.

That blindness is what motivates the more elaborate schemes: page-level write counters (expensive, generally unavailable in hardware), or sampling the guest’s own page tables to guess which pages are hot. Without them, a migration can spend its passes repeatedly resending the same small set of pages that the guest is actively using, which is the pathological case that Post-Copy Migration and Demand Paging from the Source exists to escape.