Overcommitted Guest Memory: Ballooning, Page Sharing, and Whose Reclaim Runs First

A virtual machine is assigned a memory size, and the hypervisor is not obliged to own that much memory. Guest pages are backed when the guest first touches them, and the difference between assigned and owned memory is the overcommit headroom. The mechanism that makes this work is the one described under nested paging: a guest-physical page with no second-level mapping produces a violation, and the handler decides whether to allocate a host frame, get one from a shared pool, or fail the access.

That works until it does not. Once the guest has touched most of its assigned memory, the host has committed nearly all of it, and any further pressure has to be resolved by taking pages back. The difficulty is that the guest has no visibility into the second-level tables. It believes the page is mapped because its own page table says so, and it has no way to discover that the host reclaimed the frame underneath. Every reclaim path is therefore either a mechanism for informing the guest or a mechanism for making the guest’s own reclaim run.

Host swap is the path that must not be used

The simplest reclaim is to write a guest page to host swap and invalidate the second-level mapping. The guest is untouched and completely unaware. The next access to that page produces an EPT violation, the handler reads the page back, installs the mapping, and resumes the guest.

The performance characteristics are the worst of any option. The guest suffers a page fault whose service time includes a host swap read, so the latency the guest observes is disk latency rather than the memory latency its own page-fault path was designed for. Worse, the guest’s own reclaim heuristics are based on memory latency and are therefore wrong: the guest will not avoid touching the page because from its perspective the page is resident. A host that swaps guest memory produces a tail latency distribution dominated by the moments when a guest touches a page that a host-side heuristic decided was cold.

This is why hypervisors treat host swap as a last resort and prefer mechanisms that make the guest do the reclaiming. The guest has the information the host lacks, because the guest knows which of its own pages are in active use.

The balloon: asking the guest to give pages back

The balloon driver is a paravirtual device with a guest-side driver and a host-side controller. When the host wants memory back, it tells the balloon to inflate. The guest driver responds by allocating pages inside the guest and reporting them, in batches, through the device’s queue. The host then removes the second-level mappings for the reported guest-physical pages and returns the underlying frames to its free pool. The guest’s own page tables still point at those guest-physical pages, but the pages are gone from the host’s perspective, and the guest’s allocator has removed them from its own free lists.

The guest experiences inflation as a loss of memory. Its kernel now has fewer pages available and its own reclaim machinery engages: it drops clean page-cache entries, writes back dirty ones, and in the worst case swaps anonymous pages to its own virtual disk. That is the desired outcome. The guest’s reclaim is informed by the guest’s access patterns, and it acts on the memory the guest actually considers least valuable, which is a decision the host could not make.

Deflation reverses the process. The host returns frames to the balloon, the guest driver frees the pages it was holding, and the guest’s allocator gains them back. Deflation is not always possible immediately: if the host has given the frames to another virtual machine, it must first reclaim them from that one.

The failure modes are specific. A guest without the balloon driver installed cannot be ballooned at all, because there is nothing to report pages. A guest whose driver is present but whose kernel is under its own memory pressure will fail to allocate, and the balloon stalls; the host must either retry or escalate. And an over-inflated balloon can push the guest into pathological behaviour, where the guest’s reclaim thrashes and the guest’s own swapping consumes I/O bandwidth that the host was trying to conserve. The balloon rate is a control-loop parameter, and a controller that inflates faster than the guest can reclaim productively converts a memory shortage into an I/O storm.

A later refinement in the same device is free page reporting. Instead of the host asking for a specific quantity, the guest reports pages it has already decided are free, and the host removes those mappings without any further guest involvement. This is cheaper than ballooning because the guest is not forced to allocate anything, and it recovers the memory the guest itself considers worthless.

Page sharing: finding identical content

The other class of mechanism does not need the guest’s cooperation, because it operates on content. Two guests running the same operating system have thousands of identical pages of code, and a guest’s own process may hold many identical zero pages or duplicated data structures.

The kernel’s samepage merging facility scans anonymous pages, computes a checksum over each, and keeps a tree of pages indexed by that checksum. A candidate page that matches an existing entry is compared byte for byte, and if the comparison succeeds, all mappings of the matching page are replaced with a mapping of the single retained copy, which is marked read-only. The pages are now shared.

The retention rules matter for performance. When either guest writes to a shared page, the write must not corrupt the other’s copy, so the hypervisor takes an EPT violation for the write, allocates a private copy, copies the contents, and installs a writable mapping for the writer. This copy-on-write break means that a page which is written even once after merging costs a fault, an allocation, and a copy, on top of the memory that merging was supposed to save. Pages that are genuinely identical and read-only, such as shared libraries and kernel text, stay merged. Pages that happen to be identical at one instant and diverge later are a net loss.

The scanner has tunables exposed through sysfs. A pages_to_scan value controls how many pages the scanner examines per pass, and a sleep_millisecs value controls the pause between passes, so the scanner’s CPU cost is directly settable. Statistics such as the number of shared pages and the number of full scans show whether the mechanism is finding anything. A scanner that is running too fast consumes CPU for pages that will never match; one that is too slow leaves transient duplicates merged only after they have already diverged.

# Kernel samepage merging exposes its state per-host, not per-guest.
for f in pages_shared pages_sharing pages_unshared pages_volatile full_scans; do
  printf '%s=%s\n' "$f" "$(cat /sys/kernel/mm/ksm/$f)"
done

pages_sharing counts mappings that point at merged pages, and pages_shared counts the merged pages themselves; the ratio is the effective saving. pages_volatile counts pages the scanner found to be changing too rapidly to be worth tracking, which is a signal that the workload is writing more than it is duplicating.

The deduplication that costs nothing: shared image content

The most effective sharing in practice is not memory deduplication at all. Virtual machines created from the same base image read the same file blocks, and the host’s page cache holds one copy of each distinct block regardless of how many guests reference it. The sharing is a side effect of the caching design rather than a deliberate merge, and there is no copy-on-write to break because the blocks are read-only to the guests.

This is why the choice of guest storage matters to memory performance. Guests backed by a copy of an image, where each guest has its own full file, cannot share blocks at all, because their disk contents diverge at the filesystem level even where the data is identical. Guests backed by a shared read-only base with per-guest writable overlays share the base entirely. The saving is proportional to how much of the guest’s working set comes from read-only image content, which for a freshly booted guest is most of it.

The corollary is that memory demand grows as guests write. Every guest modification becomes private, in the guest’s memory, in the writable overlay on disk, and in the host’s page cache. A fleet of guests running the same mostly-read workload converges to a much lower footprint than the same fleet running divergent workloads.

Sharing as a disclosure channel

A page that is merged becomes writable only after a fault that the hypervisor takes. The timing of that fault depends on whether the page was merged, and whether it was merged depends on whether another process holds identical content. An observer that can measure the latency of a write to a page can therefore learn whether the page’s content matches a page elsewhere in the system. That turns deduplication into a cross-tenant disclosure mechanism, which is why scanning can be restricted, why sharing can be disabled across trust boundaries, and why the write-fault latency is treated as a resource that must be accounted for rather than only saved.

Reading the memory state of a guest

The guest’s view and the host’s view of memory usage diverge, and both must be consulted. Inside the guest, free memory, page-cache size, and swap usage describe what the guest thinks it has. On the host, the assigned size, the resident size, and the balloon target describe what the host is actually spending. The gap between assigned and resident is the overcommit headroom, and it shrinks as the guest touches pages.

The gap between resident and the guest’s own free count is the pages the host has reclaimed without the guest reallocating them, which is the state ballooning is designed to make explicit. A host that reclaims without informing the guest is running on the host swap path by accident, and the guest’s measured latency will show it. The instruments for seeing that from inside the guest are the ones discussed under steal time and scheduling, which record time when the guest wanted to run but could not, and the memory analogue of that signal is the guest’s own fault rate against a host that is not keeping up.