Large Pages: What 2 MiB Granularity Buys and What It Breaks

A translation-cache entry covers one page. For a workload whose footprint is measured in gigabytes, that means a miss per 4096 bytes, and the walk behind each miss touches tables that are themselves competing for cache. The obvious fix is to make the entry cover more memory, and every mainstream architecture provides a way to do it. The fix is more constrained than it looks, because the larger unit must be contiguous in physical memory, and physical memory is not naturally contiguous at that scale.

Two Ways to Enlarge the Mapping

The architecture-level mechanism is a leaf entry at a higher level of the page-table hierarchy. On x86-64, a third-level entry can point directly at a 2 MiB frame instead of at a fourth-level table, and a second-level entry can point at a 1 GiB frame. The walk terminates early, the tables below are never allocated, and the resulting translation is cached as a single entry covering the whole range. ARMv8 has the same structure with its own size choices, and the general shape, a leaf at an intermediate level, is common.

The operating-system-level mechanism is the policy that decides when to use it. There are two distinct approaches in wide use, and they differ in who is responsible.

Explicit reservation: a pool of huge frames is set aside at boot or by an administrator, and programs request them through a dedicated interface. The pages are guaranteed to exist and are never subject to reclaim or migration. The cost is that the pool size is fixed, memory in the pool is unavailable to ordinary allocation even when unused, and a program that needs more than the pool holds fails rather than degrading.

Transparent promotion: the kernel monitors ordinary mappings and collapses a range of contiguous 4 KiB pages into a single large page when it can, splitting it back when it needs to. The program is unaware, the memory remains reclaimable, and the mechanism can be disabled per mapping. The cost is that collapse requires physical contiguity, which may not be available, and that the collapse and split operations themselves consume time.

Contiguity Is the Real Constraint

A 2 MiB page must occupy 2 MiB of physically contiguous memory, aligned to 2 MiB. Physical memory is allocated in 4 KiB frames, and after a machine has been running for a while those frames are scattered: the allocator’s free lists hold isolated frames between live allocations, and no run of 512 aligned frames exists.

The kernel’s answer is compaction. It migrates movable pages out of a candidate 2 MiB region into free frames elsewhere, updating every page-table entry that referenced them, until the region is entirely free and can be handed out as one large page. This is why the reverse mapping described in finding every mapping of a frame is a prerequisite for large pages rather than an unrelated bookkeeping detail: without a way to find all the mappings of a frame, the kernel cannot move it.

Compaction has limits. Pages that cannot be moved, because they are pinned by a device driver for DMA or locked for direct I/O, act as obstacles. A region containing one pinned frame cannot be compacted at all, so a workload that pins many scattered buffers progressively fragments the address space against large pages. This is a real tension between device I/O and memory management, and it is why systems that do a lot of unpinned DMA tend to have more success with transparent promotion.

A 1 GiB page raises the constraint by a factor of 512. Contiguity at that scale is rare on a long-running general-purpose machine, which is why 1 GiB pages are typically available only from an explicitly reserved pool, and why they are used by databases and hypervisors that can plan their memory layout rather than by ordinary processes.

What the Translation Cache Gains

The arithmetic is stark. A translation cache with a few hundred entries covers a megabyte or two with 4 KiB pages and hundreds of megabytes to a gigabyte with 2 MiB pages. A program that streams over a 4 GiB region performs on the order of a million walks with small pages and a few thousand with large ones. The difference is not a constant factor on the walk cost; it is a change in whether the walk cost appears at all.

The page-table savings are comparable. A 2 MiB region described by one large-page entry needs no fourth-level table, saving a 4096-byte allocation and 512 entries that would otherwise be initialized to not-present. For a process that maps gigabytes, the tables for the whole range shrink from tens of megabytes to a handful of entries at the upper levels. Less table memory means less cache pressure from the tables themselves, which is a second-order benefit that matters for exactly the workloads that use large pages.

The access itself also becomes cheaper in a subtle way. With a large page, the entire 2 MiB range has one set of permission bits, so the processor’s permission check and the fault path are simpler. A TLB entry for a large page may also be usable for a longer time without invalidation, since a single entry covers a range that would otherwise need 512 separate invalidations when the mapping changes.

The Waste Inside a Large Page

The cost is that the granularity of allocation becomes 2 MiB. A program that touches one byte in a collapsed region has committed 2 MiB of physical memory to it. If the same region had been left as small pages, the same byte would have cost 4 KiB.

The severity depends on the allocation pattern. A heap that grows to hundreds of megabytes and is densely used wastes almost nothing. A process with many small, sparsely used mappings wastes almost everything, and transparent promotion policies therefore restrict themselves to regions that look like they will be used densely, or to mappings the program has explicitly marked as eligible. A common default is to promote only mappings large enough to be worth it and to leave the rest alone, which is a heuristic that fails for programs whose large mappings are sparse.

The waste is bounded by the fact that the pages in a large page are not lost; they are simply all resident. A partially used large page still holds data that could be evicted, but it cannot be evicted partially. Reclaiming a single 4 KiB page from the middle of a large page requires splitting the large page back into small pages first, which means allocating a fourth-level table, populating 512 entries, and invalidating the large-page translation on every processor that holds it. That operation is expensive enough that kernels avoid it, and a large page under memory pressure can therefore pin up to 2 MiB of otherwise reclaimable memory.

Splitting is also what happens when the mapping’s properties change in a way that the large page cannot express. Changing the protection of a subrange, unmapping part of the range, or writing to a page that is shared copy-on-write all force a split, because each requires per-page differences that a single entry cannot represent. The result is a policy question: promote aggressively and pay for splits, or promote conservatively and forgo the translation savings.

Behaviour Under Pressure and Across the Interface

Large pages interact awkwardly with reclaim. The clock hand described in the replacement algorithm operates on frames, and a large page is a single reclaimable unit of 512 frames. Evicting it discards 2 MiB of resident data at once, which may include pages that were about to be used. Keeping it retains 2 MiB that may be mostly cold. Neither choice is good, and the resolution is usually to split before reclaiming, which reintroduces the split cost at the worst possible moment.

The interface exposes the distinction in ways that leak into programs. Explicit huge-page allocation returns memory aligned to the large size, so a program can rely on the alignment for its own internal mapping, but it also cannot return the memory in smaller units. A mapping created with the transparent mechanism may be promoted or not depending on system state, so a program cannot depend on the large page existing; it can only measure whether it happened. The measurement is available through counters of large-page allocations and splits, and it is the only reliable way to know whether the policy is helping for a given workload.

Virtual machines are the heaviest users, because a guest’s memory is a single large mapping in the host and the guest’s own page tables sit behind it. The translation cost for the guest is doubled by the nesting described in two-dimensional translation for virtual machines, and large pages reduce both dimensions at once. The same contiguity constraint applies more sharply there, since the host must find 2 MiB of contiguous frames for a guest page that the guest itself believes is contiguous, and a host that cannot will fall back to small pages with a measurable translation penalty that the guest cannot see or control.