Page Table Mirroring Between CPU and Device

A device driver that wants a device to access a process’s memory has a problem the CPU does not: the CPU’s page tables are the authoritative record of where the process’s pages are, and the device has its own tables, and the two must agree. When they agree, a device access resolves to the right frame. When they disagree, the device reads a frame that has been freed, migrated, or reassigned, and the corruption is silent because nothing in the system checks.

The kernel’s answer is a subsystem that lets a driver mirror a range of a process’s page tables into the device’s, with a contract for keeping the mirror honest. The contract is the interesting part. A mirror that is established once and never revised is worse than no mirror at all, because the CPU is free to change the mapping at any time and the device will not know.

The snapshot operation

The core primitive takes a description of a virtual range, walks the process’s page tables for that range, and fills in an array describing each page: its physical frame, whether it is present, whether it is writable, whether it is dirty. It does not pin the pages in the sense that a long-lived DMA mapping does. It returns a snapshot, and the caller must use it promptly.

The walk handles the cases a device would otherwise have to reimplement: anonymous pages that have not been faulted in yet, pages that are present but in a special state, huge pages that must be split or that can be passed through whole, and pages that are shared or copy-on-write. For an anonymous page that has never been touched, the walk can allocate and fault it in, which is what allows a device to take the first access to a lazily allocated buffer without the CPU having to pre-fault it.

The snapshot is explicitly not a stable mapping. Between the walk and the device’s use of the result, the CPU can write to a page, causing copy-on-write; the kernel can migrate a page for NUMA reasons; reclaim can unmap it; mprotect can change its permissions. Any of these invalidates the snapshot, and the driver has no way to detect it by inspecting the result. The detection mechanism is separate and has to be armed before the walk.

The invalidation contract

A driver registers an interval notifier covering the virtual range it cares about. The notifier receives a callback whenever the range’s mappings change in a way that matters: an unmap, a permission change, a migration, a write-protection. The callback runs in the context of whatever operation caused the change, and its job is to tell the driver to stop using the device mappings for that range and to block until the device has actually stopped.

That blocking is the hard part, and it is why the interface is shaped the way it is. The CPU operation that triggered the invalidation cannot complete until the device has stopped accessing the pages, because otherwise the CPU might free a frame the device is still reading. So the callback must wait for the device to acknowledge, which means the driver must be able to quiesce device access to a range by waiting for outstanding work, by invalidating the device’s translation cache, and by preventing new work from starting on that range.

The ordering requirement creates a lock-order problem. The invalidation typically runs while the kernel holds the affected address space’s mmap lock, and the driver must not, from inside the callback, take a lock that a thread holding that lock might need. Drivers that do so deadlock, and the deadlock is only reachable under memory pressure or during a migration, which makes it rare and hard to reproduce.

The sequence a driver follows is therefore: register the notifier; on each use, take the snapshot; program the device with the result; use it; on invalidation, quiesce and release. The snapshot and the invalidation are paired, and the driver must not assume the snapshot remains valid after the callback has fired, even if the callback fired for a different range, since invalidation granularity can be coarser than the range the driver registered.

/* Sketch of the driver-side ordering for the kernel's HMM notifier
   interface; device_quiesce and program_device_tables stand in for
   driver code. Locking is omitted. */
struct region {
    struct mmu_interval_notifier notifier;
    struct hmm_range range;     /* start, end, and the filled pfn array */
    struct mutex lock;          /* held while a snapshot is in use */
    bool valid;
};

static bool on_invalidate(struct mmu_interval_notifier *n,
                          const struct mmu_notifier_range *range,
                          unsigned long cur_seq)
{
    struct region *r = container_of(n, struct region, notifier);

    /* Block until the device has stopped touching the range, then
       drop the mirrored mappings. The CPU side is waiting on us. */
    device_quiesce(r);
    r->valid = false;

    /* Tell the core this invalidation has been observed. */
    mmu_interval_set_seq(n, cur_seq);
    return true;
}

static int mirror(struct region *r, struct device *dev,
                  unsigned long start, unsigned long end)
{
    r->range.notifier = &r->notifier;
    r->range.start = start;
    r->range.end = end;

    /* Walk the CPU page tables and collect frames for the device. */
    if (hmm_range_fault(&r->range) < 0)
        return -EFAULT;

    program_device_tables(dev, &r->range);
    r->valid = true;
    return 0;
}

Migrating a page to device memory

The same subsystem supports the reverse direction: moving a page out of the CPU’s page tables and into memory that belongs to a device. The mechanism replaces the process’s page table entry with a special entry that records where the page went, so a later CPU access faults and the kernel knows to bring it back. The device, meanwhile, has the page in its own memory and a mapping that resolves to it.

The migration runs in stages. The first stage walks the range, write-protects the pages it intends to move, and collects them without yet changing anything, so a failure part-way leaves the mapping intact. The second stage installs the new entries and copies the data. The third stage releases the old pages and drops the temporary references. The staging exists because the copy can fail or be interrupted, and a migration that has already destroyed the source cannot be rolled back.

Device memory used this way is registered as a special zone rather than as ordinary memory. Pages in it are not on the kernel’s free lists, are not reclaimable by the ordinary page reclaim path, and are not visible to the buddy allocator. A CPU access to a page that has been migrated there produces a fault that the driver handles by migrating it back, which is why the fault path and the migration path are the same code reached from opposite directions.

The consequence for the device driver is that it must implement both directions. A driver that migrates pages out but cannot service a fault that brings them back leaves the process with pages it can never touch again. The fault handler runs in the context of the faulting thread, which means it can sleep, allocate, and take locks, but it must not deadlock against the invalidation callback, since the two can be invoked concurrently for adjacent ranges.

Three guarantees the mirroring does not provide

Three things are commonly assumed and are not provided.

The first is coherence. Mirroring establishes that the device and the CPU agree on which physical frame a virtual address refers to. It says nothing about caches. If the device has a cached copy of a line and the CPU writes to that line through its own cache, the device’s copy is stale unless the platform implements hardware coherence or the driver performs cache maintenance. The invalidation callback fires for a write-protection or a mapping change, not for every CPU store; a store that leaves the mapping intact is invisible to the mirroring machinery.

The second is atomicity at a granularity the device can use. The snapshot describes pages, and a page is 4 KiB or larger. A device that needs to observe a consistent multi-page view of a structure whose fields span pages has to establish that consistency itself, with the ordering machinery of system-scope atomics and memory ordering.

The third is performance. The walk is a page table traversal with a cost per page, and the invalidation callback is a synchronization point with the device. Neither is free, and both scale with the range size. A driver that re-walks a large range on every operation spends its time in the walk; the usual fix is to walk once and keep the mapping until an invalidation actually arrives, rather than re-validating defensively.

Where the design pays off and where it costs

The payoff is that a device can access any of a process’s memory without the application having to register buffers in advance. That is what makes a managed-memory programming model possible at all: the runtime is a driver, the driver uses this machinery, and the application’s ordinary allocations become device-accessible without any explicit call. The faults described in GPU page faults and demand migration are serviced by exactly this path.

The cost is that the kernel is now in the device’s fault path. Every device fault that needs a CPU page table consulted goes through the kernel, which means a context switch, a lock, and a page table walk. On a system where the fault rate is low, this is invisible. On a system where the fault rate is high, as with scattered access, small pages, or a working set that does not fit, the kernel becomes the bottleneck and the device’s own throughput is irrelevant.

The other cost is complexity in the driver, and it lands on the invalidation path. Every operation that can change a mapping must be considered: fork, mremap, mprotect, MADV_DONTNEED, reclaim, migration, userfaultfd handling, and the exit path. Missing one produces a mapping the device believes is valid and the CPU has already changed, which is the silent corruption the subsystem exists to prevent. The correctness argument is not local to the driver; it is a property of the driver’s interaction with every part of the memory management code.

This is why the machinery is used by a small number of drivers and why the interfaces have changed shape over successive kernel releases. The abstraction boundary between “the kernel owns the page tables” and “the device has its own” is genuinely difficult, and the design that resolves it is a compromise: the kernel remains authoritative, the device mirrors, and the mirror is only as good as the invalidation path that maintains it.

The alternative designs are worth naming for contrast. Pinning gives a stable mapping with no invalidation machinery, at the cost of making the memory unmanaged, as covered in pinned host memory and asynchronous transfers. Mapping host memory permanently into the device gives a stable mapping with no kernel involvement in the fault path, at the cost of never being able to move the data, covered in host-mapped memory over PCIe. Mirroring is the design that keeps the memory movable, and it pays for that with the invalidation contract.