Encoding Guest Memory for the Migration Stream

A migration stream carries the guest’s memory image plus device state. Raw, that is the guest’s full RAM: a 16 GiB guest produces 16 GiB on the wire, and at 10 Gbit/s that is about 13 seconds of pure transfer, before any dirty pages are retransmitted. Most of those bytes are not information. An idle or lightly used guest is dominated by pages that are entirely zero, and a busy guest’s pages compress well because they contain pointers, structure padding, and repeated patterns.

The encoding layer is what closes the gap between the image size and the bytes that must cross the link. It sits between the dirty-page bookkeeping and the transport, and its design choices determine how much CPU the source spends, how much the destination spends decoding, and how a bad guess about the workload turns into either wasted bandwidth or wasted cycles.

Sparse detection before compression

The cheapest transformation is the one that sends nothing. A page that reads as all zeros is skipped entirely and recorded in a bitmap of skipped pages that the destination uses to zero-fill. Testing 4 KiB for zeros is a handful of vector compares — on a machine with 512-bit registers, 64 comparisons of 64 bytes each, which is a few dozen cycles per page. At that cost, scanning the whole image is negligible compared with transferring it.

The saving is workload-dependent and can be extreme. A freshly booted guest, a guest that has freed a large heap, and a guest whose allocator touches only the first bytes of each page all produce mostly-zero images. A guest running a JVM after a full garbage collection is a different story: the heap is mostly live and mostly non-zero.

Zero detection has a correctness trap that is easy to get wrong. A page must be tested at the moment it is sent, not at the moment it was recorded as dirty, because the guest may have zeroed it since. Testing at send time costs the scan again but removes the race. The inverse race — a page that was zero at scan time and is written before the destination zero-fills it — is handled by the dirty bitmap: the write marks the page dirty for the next pass, so the destination’s zero-fill is overwritten by the later real content.

The destination must therefore treat the skip bitmap as authoritative only for the pages in that pass, and must not assume a skipped page stays zero.

Choosing a compressor by page population

Once zero pages are removed, the remaining pages fall into populations with very different compressibility:

• Text and code pages: highly repetitive, small alphabet, compress well at any ratio setting.
• Heap pages with pointer-rich structures: moderate compression; the high bytes of pointers repeat, the low bytes do not.
• Already-compressed content (images, encrypted volumes, compressed archives inside the guest): incompressible, and running a general compressor over them costs CPU for no gain.
• Page tables and kernel structures: small and highly structured, compress extremely well.

A single compression setting applied to all of them is a compromise. A fast compressor such as LZ4 will find the easy wins at a few hundred megabytes per second per core; a heavier one such as zlib’s DEFLATE at a high level will find more but can drop below tens of megabytes per second, which is slower than the link it is trying to feed. The crossover is a function of link speed: at 1 Gbit/s (about 119 MiB/s) a compressor that runs at 100 MiB/s is already the bottleneck; at 25 Gbit/s it is not.

The decision that matters is whether compression is enabled at all. Sending raw is correct when the link is the constraint and the source CPU is not, or when the pages are known incompressible. Sending compressed is correct when the link is slow and the source has idle cores. There is no setting that is right for every guest, which is why the feature is a knob rather than a default.

Framing overhead is a real fraction at small page sizes

Each page sent needs a record: which guest physical address it belongs to, how it is encoded, and how long the payload is. The minimum viable record is a few bytes; a versioned format with alignment padding can reach tens of bytes. At 4 KiB pages, 32 bytes of header is under 1 percent, which is fine. At 4 KiB pages with per-page compression and a 64-byte header, the overhead approaches 2 percent, and the compressor’s own frame headers add more.

The counter-move is batching: concatenate several pages into one record with a run-length address encoding, so a run of consecutive dirty pages costs one header plus a length. Consecutive dirty pages are common in guests that allocate sequentially, and rare in guests with fragmented heaps. Batching also improves compression ratio, because the compressor sees a larger window and can match across page boundaries — but only if the pages are contiguous in the stream, which requires the sender to sort or group them, adding bookkeeping.

The general shape of the trade: smaller records reduce latency to first byte and make the stream easier to resume after an interruption; larger records amortize framing and compress better. Implementations expose this as a page-size or batch-size parameter rather than picking a value, because the right choice depends on the guest’s allocation pattern.

/* Sketch of per-page record framing; the header cost is the point. */
struct page_record {
    uint64_t gpa;        /* guest physical address of the page */
    uint32_t length;     /* bytes of payload following */
    uint8_t  encoding;   /* RAW, ZERO, LZ4, ZLIB */
} __attribute__((packed));

static size_t encode_page(struct page_record *rec, const void *page,
                          size_t page_size, void *out, size_t out_len)
{
    if (is_zero_page(page, page_size)) {   /* cheaper than sending */
        rec->encoding = ZERO;
        rec->length = 0;
        return sizeof(*rec);
    }
    size_t n = lz4_compress(page, page_size, out + sizeof(*rec), out_len);
    if (n == 0 || n >= page_size) {        /* incompressible: send raw */
        memcpy(out + sizeof(*rec), page, page_size);
        rec->encoding = RAW;
        rec->length = page_size;
        return sizeof(*rec) + page_size;
    }
    rec->encoding = LZ4;
    rec->length = n;
    return sizeof(*rec) + n;
}

The fallback to raw is not an optimization detail; it is what prevents compression from making a stream larger. Every compressor has incompressible inputs where the output exceeds the input by a small framing overhead, and a sender that always compresses pays that penalty on exactly the pages that were already dense.

Self-comparison and the limits of delta encoding

A more aggressive scheme compares each page against its previous version and sends only the differing bytes. This can beat compression on pages that change slightly between passes: a page containing a data structure where one field was updated has almost no entropy in its difference.

The costs are substantial. The source must retain the previous version of every page, doubling the memory footprint of the migration, or read it back from the destination, which converts a local comparison into a network round trip. The destination must apply the delta to the correct base version, so the stream carries version identifiers and the two sides must agree on which base each delta applies to. And the comparison itself is a full page read plus a byte-wise diff, which is not cheaper than a fast compressor over the same bytes.

In practice, delta encoding pays off for guests whose pages change in small, localized ways — a page table, a journal header, a reference-count array — and loses to plain compression for guests whose pages are rewritten wholesale. Since a migration cannot know which population it is dealing with before it starts, the scheme is usually reserved for a second phase after the first pass reveals the pattern.

What encoding cannot fix

Compression and sparse detection reduce the bytes per pass, but they do not change the convergence ratio. If the guest dirties memory faster than the link can carry the compressed stream, the migration still diverges. A compressor that achieves a 4:1 ratio on a workload effectively quadruples the link rate for that workload, which can convert a diverging migration into a converging one, but only if the dirty pages are compressible. A guest writing an incompressible, page-filling stream at 2 GiB/s cannot be rescued by encoding at any ratio.

There is also an interaction with the freeze. The residual pages copied inside the stop-and-copy window are compressed too, so the compressor’s throughput becomes part of the downtime, not just part of the total time. A slow compressor that is fine during pre-copy, where it overlaps with guest execution, becomes a direct multiplier on downtime when it runs inside the freeze. The encoding parameters chosen for the bulk passes are therefore not automatically the right ones for the final pass, and implementations commonly use a faster path for the residual.

Encoding is the layer that makes the numbers in Dirty Page Rate and the Convergence Condition better than the raw arithmetic suggests, but it operates on the same dirty bitmap produced by Hardware Dirty-Bit Tracking for Guest Memory and inherits its page granularity. It cannot recover the bytes that were dirtied by a single store into a large page; it can only make those bytes cheaper to carry.