Kernel Page Table Isolation and What It Costs on Every Trap

For most of the history of paged operating systems, the kernel was mapped into every address space. A user process’s page tables contained the kernel’s code and data at high addresses, marked supervisor-only so that user-mode code could not read them but present so that a trap did not need to change the page-table root. The arrangement made system calls cheap: the transition from user to kernel changed the privilege level and nothing else about translation.

That design has a flaw that became visible when speculative execution was studied closely. A processor may perform a load whose result is later discarded because the privilege check fails, and the discarded load can still leave an observable trace in the cache. If the kernel is mapped in the current address space, even as supervisor-only, a speculative load can reach it and the trace can be measured from user mode. Removing the kernel from the user address space removes the reachable target.

Two Roots Instead of One

Kernel page-table isolation keeps two complete page-table hierarchies per process. The user hierarchy maps the process’s user memory and a minimal set of kernel text needed to enter and leave the kernel: the entry trampolines, the interrupt descriptor table, and the per-CPU data the entry path touches before it can switch roots. The kernel hierarchy maps everything the kernel needs, including the full kernel image, and also the user mappings, because the kernel must be able to copy to and from user buffers.

The root register points at the user hierarchy while user code runs and at the kernel hierarchy while kernel code runs. The trap entry sequence therefore has a problem: it begins executing at an address that must be mapped in the user hierarchy, and its first task is to switch the root register to the kernel hierarchy. The trampoline that performs this must not touch anything that is only in the kernel hierarchy, and it must save the user root somewhere it can find it after the switch, which is why the per-CPU area holding that value is mapped in both.

/* The shape of the entry trampoline, reduced to the essential steps. */
__attribute__((naked)) void entry_trampoline(void)
{
    __asm__ volatile(
        "swapgs\n\t"              /* obtain the kernel per-CPU pointer */
        "movq %%rsp, %%gs:0\n\t"  /* stash the user stack pointer */
        "movq %%gs:8, %%rsp\n\t"  /* load the kernel stack */
        "movq %%cr3, %%rax\n\t"
        "movq %%rax, %%gs:16\n\t" /* save the user root */
        "movq %%gs:24, %%rax\n\t" /* kernel root, recorded per CPU */
        "movq %%rax, %%cr3\n\t"   /* switch to the kernel hierarchy */
        "jmp  kernel_entry\n\t"
    );
}

The real sequence is longer and handles the interrupt stack, the register save area, and the case where the trap arrived from kernel mode. The structure is what matters: a few instructions that touch only memory present in both hierarchies, followed by a root switch after which everything is available.

The reverse transition happens at return. Before the kernel executes the return instruction, it switches the root back to the user hierarchy and invalidates the entries that must not survive the switch. Doing this before the return rather than after is essential; the return instruction itself fetches from the user hierarchy’s trampoline page.

The Invalidation Problem

A root switch does not automatically invalidate cached translations. On x86-64, writing CR3 with the no-flush bit clear flushes translations that are not tagged as global; entries marked global survive. If the kernel’s translations were allowed to survive into user mode, the isolation would be pointless, because a speculative access could still hit a cached translation and proceed to the physical frame without consulting the tables at all.

The isolation design therefore marks nothing global that must be hidden, and it relies on the root switch flushing the rest. That flush is the expensive part: after it, the kernel starts with an empty translation cache for its own mappings, and the user process starts with an empty cache for its mappings if the switch to the user root also flushes. Architectures with tagged translations can avoid the second flush by giving each root a distinct address-space identifier, which preserves the user’s entries across the kernel’s execution, and designs differ on whether that is available.

Without tagging, every system call, every interrupt, and every fault costs a full translation flush in both directions, which is the same invalidation requirement described in the translation cache and its misses, applied at the coarsest possible granularity. The user process’s translation cache is empty when it resumes, so its next accesses walk the page tables again. A program that makes many cheap system calls, such as a server reading a socket in a loop, pays this on every call, and the cost is not the call itself but the reconstruction of translation state afterwards.

Where the Cost Actually Lands

The cost of isolation is visible mainly in workloads that cross the boundary often. A system-call-heavy workload pays a translation rebuild per call; a compute-bound one pays it a handful of times per scheduling interval and does not notice. The magnitude depends on the processor generation, on whether translations are tagged, and on how many distinct pages each side touches between transitions, so the useful statement is about the mechanism rather than a percentage: cost is proportional to transitions multiplied by pages touched across each transition.

Three mitigations reduce it, and each has a limit.

Keeping the kernel’s frequently used mappings in the user hierarchy narrows the flush’s effect. Only the entry trampoline and per-CPU data are there, so the user hierarchy’s tables are small and cheap to populate after a flush. The kernel hierarchy, by contrast, has a large footprint, and its translations are discarded on every entry. The kernel therefore experiences a translation miss on its first access to each of its own pages after every transition, which is why entry paths are written to touch few distinct pages.

Translation tagging, where available, removes the user-side flush. The user root keeps its identifier across the kernel’s execution, so the user’s entries survive and the return path only has to invalidate the entries belonging to the kernel’s identifier, or none at all if the kernel’s entries are tagged separately. This converts the cost from “flush everything twice” to “flush the kernel’s own entries once”, and it is why processors with tagging show much smaller isolation overhead.

Reducing transitions helps but is not always possible. System calls that can be served without entering the kernel at all, through a shared-memory fast path, avoid the flush entirely, and this is the motivation for some of the vDSO mechanisms that serve clock reads from user space. The approach works for a narrow class of operations and does nothing for the rest.

Interaction With the Rest of the Memory System

Isolation makes the kernel’s translation behaviour asymmetric with the user’s. The kernel’s page tables are the same across all processes except for the user mappings they include, so the kernel’s own mappings could in principle be shared. They are not shared in the isolation design, because each process has its own kernel hierarchy, which multiplies the memory cost of page tables by the number of processes. The upper levels of the kernel hierarchy are typically shared between processes to bound this, with the lower levels private where they describe per-process user memory.

The interaction with large pages is favourable. A kernel mapping that uses a 2 MiB page needs one translation entry instead of 512, so the flush after entry destroys one entry rather than 512 and the rebuild costs one walk rather than 512. This is a case where the large-page machinery described in large pages and translation reach pays for itself twice: once in ordinary translation reach and once in reduced isolation overhead.

The interaction with the fault path is more delicate. A page fault taken in user mode enters through the trampoline and switches to the kernel root, and the handler then works on the user hierarchy, whose root address it must retrieve from the per-CPU data. Writing a page-table entry into the user hierarchy while running under the kernel root is possible because the kernel hierarchy maps the user’s page-table pages as well; the kernel’s direct map of physical memory covers them. The invalidation must then be issued for the user’s address space, not the current one, which is why the invalidation instructions take an address-space identifier or require a temporary switch.

What Isolation Does Not Cover

The mechanism removes the kernel from the user’s reachable address space. It does not remove the kernel’s effect on shared microarchitectural state. Branch predictors, cache sets indexed by physical address, and prefetchers are shared between privilege levels, and a speculative access that never completes can still influence them. Isolation raises the cost of the attack that motivated it without closing the class.

It also does not protect against a kernel bug that copies user-controlled data into a kernel address, nor against a fault handler that returns to user mode with a stale root. Those are implementation failures rather than design limits, and they are the reason the entry and exit paths are written in assembly with the smallest possible surface and are audited as a unit.

The broader lesson is that address-space layout is a security boundary, not only a performance structure. Once the kernel shares a translation hierarchy with user code, every supervisor-only mapping is a potential speculative target, and the cost of separating them is paid on every transition for the lifetime of the system. The design decision is therefore made once, at the level of the whole kernel, and the rest of the memory manager has to accommodate it.