Cacheability, Write-Combining, and Mapping Device Memory

Everything described so far assumes that memory behaves like memory: reads return what was last written, writes eventually become visible, and the processor is free to reorder and cache accesses within the rules of its memory model. Device registers mapped into the address space break every one of those assumptions. A read of a status register can clear an interrupt. A write to a control register can trigger a transfer. Two writes to adjacent registers may need to arrive in a specific order, and the device may not tolerate a speculative read of an address that has side effects.

The page table is where the processor learns that a particular range must be treated differently. The permission bits determine who may access a page; the attribute bits determine how the access is performed, and for device memory the attribute bits are the difference between a working driver and a machine that hangs on the first probe.

The Attributes and What They Control

On x86-64 the relevant control is a set of three bits in the page-table entry, historically called the page-level cache controls: one that disables caching, one that enables write-combining, and one that selects write-through rather than write-back. They combine with a register that sets a default for the whole memory type range, and the effective type for an access is derived from both. The types that matter in practice are:

Write-back, the default for ordinary memory, where a line stays in cache and is written to memory when evicted. Read and write allocate on a miss, and the processor may reorder accesses to different lines freely.

Uncached, where every access goes to memory and no line is allocated. This is the safe choice for device registers: a read reaches the device, a write reaches the device, and nothing is held in a cache that could satisfy a later access with a stale value.

Write-combining, where writes are gathered into a buffer and flushed as a burst, but reads are not cached. This is the right choice for framebuffers and for device memory that is written in bulk, because it converts many small writes into few large transactions without the read-side hazards of caching. It carries an ordering caveat: writes to the same line may be merged, and the program must not depend on their individual arrival.

ARMv8 expresses the same distinctions through memory attributes in the page-table descriptor, with device memory split into several flavours by how strongly ordered the accesses are. The names differ; the underlying taxonomy of cached, uncached, and write-combining with an ordering strength does not.

/* Map a device register page uncached. Helper names are local to this sketch. */
void *map_device_page(phys_addr_t pa)
{
    void *va = reserve_va_range(PAGE_SIZE);
    pte_t *pte = walk_to_leaf(kernel_mm, (uintptr_t)va);

    /* Present, writable, supervisor-only, no caching, no write-combining. */
    *pte = pte_mkdev(pa & PAGE_MASK,
                     PTE_PRESENT | PTE_WRITE | PTE_NX);
    flush_translation(va);               /* no stale cached translation */
    return va;
}

The flush_translation call is not optional. If the virtual range was previously mapped as ordinary cached memory, a translation describing it as cacheable may still be in the translation cache, and the processor will continue to cache device accesses until that entry is evicted. Mapping device memory over an existing mapping without an invalidation is a classic driver bug, and its symptom is a device that works until the stale entry happens to be replaced.

Why Caching a Register Is Wrong

A device register is not storage. Its value can change without any write from the processor: a status bit sets when a packet arrives, a counter advances, a ready bit clears. If the register’s page is mapped as cacheable, a read brings the line into cache, and subsequent reads are satisfied from the cached copy, so the driver polls a value that never changes. The device’s interrupt may fire and the handler may read the same stale line.

Worse, a write to a control register on a cached line may sit in the cache indefinitely, so the device never sees it. The processor believes it has issued the write, the driver proceeds to wait for a completion that cannot occur because the command was never delivered. The failure is timing-dependent and appears only when the cache line is eventually evicted for unrelated reasons, which makes it nearly impossible to diagnose from a symptom report.

Write-combining has a subtler hazard. The processor may merge adjacent writes into one transaction, which is exactly what is wanted for a framebuffer and exactly what is not wanted for a device with two registers where writing one has a side effect. If the device decodes the merged write as a single access to the first register, the second register is never written. Device memory that must be written register by register needs uncached access, not write-combining, and drivers for such devices insert barriers or use accessors that force separate transactions.

Ordering Beyond the Attributes

Attributes control caching; they do not by themselves control the order in which accesses become visible. A driver typically writes a command to one register and then writes a doorbell in another to tell the device to start. If the two writes are reordered, the device sees the doorbell before the command and acts on stale data.

The architectural answer is a barrier instruction with the right strength for the direction of the dependency. On x86, the strong ordering of ordinary memory accesses covers most cases, but device memory accessed through uncached mappings is subject to weaker guarantees, and the barrier instructions (MFENCE, and the serializing effect of certain instructions) are what the driver uses to enforce order. On ARMv8 the DMB instruction with the appropriate domain provides the ordering, and the architecture’s memory model makes the requirement explicit rather than implicit.

The correct pattern is a barrier between the command write and the doorbell write, and often a barrier after reading a status register before acting on it. The barrier is a property of the code, not of the mapping, so no page-table attribute can substitute for it. This is the part of device access that page attributes cannot fix, and it is why driver code that touches registers directly is written with explicit ordering annotations even when the mapping is uncached.

DMA and the Coherence Question

A device that writes to system memory through DMA is a second writer to ordinary cached memory, and the processor has no way to know when the device’s write lands. Two designs handle this.

Coherent DMA, on systems where the interconnect maintains coherence with the processor caches, needs no cache maintenance in software: the device’s write invalidates or updates the relevant cache lines. The driver’s responsibility is limited to ordering, ensuring that buffers are set up and their addresses communicated before the device is started.

Non-coherent DMA, common on embedded systems and on some ARM configurations, requires the driver to clean the cache before handing a buffer to the device and to invalidate it before reading data the device wrote. Cleaning pushes dirty lines to memory so the device sees current data; invalidating discards lines so the processor does not read stale cached copies of what the device wrote. Getting this wrong produces a buffer that the device filled correctly but the processor reads as old contents, and the error is silent.

The mapping attributes for a DMA buffer are therefore part of the contract. A buffer mapped as cacheable is fine for coherent DMA and requires maintenance for non-coherent DMA. A buffer mapped as uncached avoids the maintenance entirely at the cost of every processor access to it going to memory. The choice is a throughput trade, and for a buffer that the processor touches frequently and the device touches rarely, the maintenance cost may be lower than the uncached access cost.

Pinning, IOMMUs, and the Address the Device Sees

A DMA transfer needs a physical address, and the device needs that address to remain valid for the duration of the transfer. A page that the kernel might migrate or reclaim cannot be handed to a device without being pinned, which is the mechanism that makes the reverse map’s job harder, as described in reverse mapping and page migration. Pinning is per-page and per-transfer, and a driver that pins a large scattered buffer creates exactly the fragmentation that defeats large-page allocation.

An IOMMU removes the need for physical addresses in the device by giving the device its own translation, programmed by the kernel. The device then issues addresses in an I/O virtual space, and the IOMMU translates them to physical frames. This has three consequences: the device can be given a buffer that is not physically contiguous, a device can be prevented from accessing memory outside its assigned range, and the translation must be invalidated when the mapping changes, with the same rules that apply to processor translations. The two-stage composition, a device address resolved through one table into a physical frame, is structurally the same problem as nested paging for virtual machines. A driver that bypasses the IOMMU by using physical addresses gives up the isolation, which is why systems that support it often make bypass an explicit configuration rather than a default.

The Boundary

The attribute mechanism is where the virtual memory system stops being about capacity and starts being about semantics. A page-table entry that says “present, writable, supervisor” describes a permission; one that also says “uncached, strongly ordered” describes a contract with a device that the processor cannot verify and the kernel cannot enforce. Every later mechanism, from reclaim to migration, has to exclude such pages from its assumptions, and a frame that is mapped to a device register must never be treated as a candidate for eviction, compaction, or write-back. The kernel marks those frames as reserved, and the marking is the only thing standing between a memory manager’s optimization and a device that has been handed a stale address.