Ordering Stores to Persistent Memory: CLWB, SFENCE, and Failure Atomicity

A byte-addressable persistent memory DIMM sits on the memory bus. A store to it is an ordinary store instruction, and that is the whole problem: the instruction retires into a cache, and the cache is volatile. On a machine with a volatile cache, a store that has executed is not durable, and the instruction set offers no way to make it durable except by writing the containing cacheline out and then fencing. Two instructions do the writing, one does the ordering, and a third arrangement of them is wrong in a way that produces no symptom until power fails.

Three instructions and what each guarantees

CLFLUSH writes a cacheline back to memory and invalidates the copy in the cache. It is ordered with respect to other CLFLUSH instructions and with respect to stores to the same address, but it is not ordered against unrelated stores, and it is a relatively slow operation because it must invalidate. CLFLUSHOPT writes the line back and invalidates it but is explicitly weakly ordered: it may be reordered with other stores and with other CLFLUSHOPT instructions, so a sequence of them can complete out of order. CLWB writes the line back and, unlike the other two, may leave it in the cache in a clean state, so a subsequent read does not miss. It is also weakly ordered.

The ordering is supplied by SFENCE, which orders all stores and flushes that precede it against all stores and flushes that follow it, in program order. On x86, ordinary stores are already ordered against other stores, but the flush instructions are not ordered against stores by default, which is why the fence is needed. The canonical durable store sequence is therefore: store the data, flush the cachelines containing it, then SFENCE.

#include <immintrin.h>
#include <stdint.h>

static void persist_range(void *p, size_t len)
{
    /* Lines are 64 bytes here; iterate over the lines the range intersects,
     * not over multiples of 64 counted from p, which misses the ends. */
    uintptr_t start = (uintptr_t)p & ~(uintptr_t)63;
    uintptr_t end   = ((uintptr_t)p + len + 63) & ~(uintptr_t)63;

    for (uintptr_t a = start; a < end; a += 64)
        _mm_clwb((void *)a);
    _mm_sfence();   /* every flush above completes before any later store */
}

The loop’s bound is the subtlety. A range that is not cacheline-aligned touches one extra line at each end, and flushing only len bytes starting at p leaves the line containing p + len partially dirty if the range ends mid-line. The correct iteration is over the lines that intersect the range, computed from the aligned start and the aligned end, not over multiples of 64 from p.

A flush without a fence proves nothing

The failure that motivates the fence is worth spelling out. Consider a transaction that writes a data block and then a commit flag. The code stores the data, flushes it, stores the flag, flushes it. Without fences, the second store can be visible, and its flush can complete, before the first flush completes. After a crash, the flag is durable and the data is not. The recovery code reads the flag and trusts a block that holds a mixture of old and new bytes.

CLWB makes this worse than CLFLUSH did, because CLWB is weaker by design: it does not invalidate, so the line remains visible to other cores and to the flush machinery’s internal state in a way that permits more reordering. The performance gain is real: a clean line that stays cached avoids a later miss, but the ordering obligation moves entirely to the fence.

There is also a constraint that catches implementations using non-temporal stores. A streaming store that bypasses the cache still has to be ordered against the persistence of the surrounding data, and the fence that orders it is the same one. Mixing temporal and non-temporal stores in one transaction is legal but requires the fence to cover both, which means it must come after all of them rather than after the group that happens to be temporally stored.

Platforms that do not need the flush

The instruction sequence above is necessary on a platform whose cache loses power. Intel’s asynchronous DRAM refresh, ADR, uses residual energy to flush the memory controller’s write buffers into the DIMM on power loss, which makes stores that have reached the controller durable, but not stores still sitting in a CPU cache. Extended ADR, eADR, extends that domain to include the cache hierarchy, so a store that has retired is already inside the persistence domain and needs no flush and no fence.

The distinction changes what the code must do and cannot be discovered portably. A program compiled for the eADR case and run on an ADR-only platform loses data; a program that always flushes and fences is correct on both and pays the cost on the platform that did not need it. Libraries handle this by querying the platform at startup, which means the durable-store path is chosen at run time rather than by a compile-time flag. The query itself is a platform interface, not a portable one, and a program that hardcodes the assumption is betting on a hardware configuration it cannot see.

The fence has a second cost beyond its own latency. It serializes the pipeline, so a loop that flushes and fences per record turns a streaming write into a sequence of round trips. The obvious optimization is to accumulate many records, flush all their lines, and fence once — which is exactly the batching argument that applies to fsync and to log commits, and which is why a persistent data structure’s throughput is governed by how many barriers it issues per unit of work rather than by its instruction count.

Failure atomicity on a persistent heap

A durable store is the primitive; a transaction is what an application actually needs. On a persistent heap, the standard construction is undo logging: before modifying an object in place, copy the old bytes into a log record, make the log record durable, then modify. A commit makes a marker durable; recovery replays committed transactions forward or rolls uncommitted ones back by restoring the logged bytes.

/* Undo-style failure atomicity, sketched. */
log_begin();
log_record(old_bytes, sizeof(old_bytes));   /* copy before overwrite */
persist_range(log_tail, sizeof(old_bytes)); /* undo image durable */
memcpy(target, new_bytes, sizeof(new_bytes));
persist_range(target, sizeof(new_bytes));   /* new value durable */
log_commit();                               /* single durable marker */

The ordering requirement is the mirror of write-ahead logging’s, as set out in Write-Ahead Logging and the Commit Record: the undo image must be durable before the overwrite, and the commit marker after it. The difference is that the log lives in the same address space as the data and is reached by ordinary stores, so the durability of a log record is a flush and a fence rather than a write and an fsync. The reduction in cost is substantial: no system call, no block layer, no device queue. The correctness argument is unchanged, which is why the persistent-memory libraries reuse the log-structured vocabulary rather than inventing a new one.

The allocator is the part that is easy to overlook. A persistent heap must hand out space in a way that survives a crash, so its free lists and its allocation metadata have to be updated atomically with respect to the allocation. The usual technique is to write the object’s contents first, flush, and only then link it into the structure that makes it reachable; a crash between the two leaves the object allocated but unreachable, which is a leak rather than a corruption. Recovery then has to find and reclaim those leaks, and the only way to find them is to walk the reachable graph and compare it against the allocator’s records. That is the persistent analogue of the mark-and-sweep problem, with the added difficulty that the walk itself must not modify anything until it has finished.

Where the model breaks down

The instruction-level model assumes the store and the flush target the same physical memory. A persistent memory region mapped into a process’s address space through a filesystem is subject to the filesystem’s own metadata and to the mapping’s properties; a flush of a cacheline makes the data durable in the DIMM but does not make the file’s size or its block allocation durable. A program that grows a persistent memory file and flushes only the data has written bytes that belong to no file after a crash. That boundary between the byte-addressable region and the filesystem that names it is the subject of Memory-Mapped Persistence and msync.

The model also assumes the fence is enough to order the flush against everything that matters. On a platform with a volatile write buffer between the cache and the media, the fence orders the flush against later stores in the same domain but says nothing about when the buffer drains. That is what ADR exists to solve, and it is why a platform’s persistence domain has to be established before any of this code is meaningful, which is the same dependency described at the level of the block layer in Cache Flushes and FUA.

Finally, the compiler is a participant. A store that is never read again can be eliminated; a flush can be moved across a function boundary; a fence is a compiler barrier only if it is one. The intrinsics are defined to be barriers, but a volatile pointer does not order anything on its own, and a memcpy that the compiler turns into vector stores still needs the flushes to cover the actual lines written. The rule is that every durable write goes through a function the compiler cannot see into, or through an intrinsic with documented ordering, and that the flush loop is computed from the bytes that were actually stored rather than from the bytes the source code appears to store.