A Completed Write Is Not Yet Durable: Persistence Domains and Ordering

A process writes 4096 bytes with one write call, receives 4096 back, and the machine loses power a microsecond later. After the reboot the file may hold the new bytes, may hold the old bytes, or, if the write crossed a sector boundary and the device was interrupted mid-transfer, may hold a mixture of both. The return value reports that the kernel accepted the bytes into the page cache. Everything between that acceptance and the flash cell is invisible to the caller, and each layer in between is a place where a crash can strand the data.

The store path has five volatile layers above the media

A store instruction retires into the store buffer, then into L1, then L2, then the last-level cache. Those are tens of kilobytes, a few hundred kilobytes, and a few megabytes respectively on common x86-64 parts, and all three lose their contents when power goes away. Dirty lines reach memory only when the cache decides to evict them, which is a capacity and coherence decision with no relation to durability. The memory controller then forwards them to DRAM, which is also volatile, and DRAM refresh is what keeps it readable at all.

From there the write crosses a PCIe or SATA link to the device. A solid-state drive or a RAID controller typically holds a write-back cache of its own, and that cache is the layer most often forgotten. When the device reports completion it may be reporting that the data is in that cache, not that it has been programmed into NAND or written to a platter. The kernel adds one more volatile stop: the write path copies the user buffer into a page-cache page, marks it dirty, and returns. The page is 4096 bytes on x86-64, and it will be written back whenever the writeback machinery decides, possibly seconds later. O_DIRECT skips the page cache but not the device cache, so it changes latency and memory pressure rather than the durability question.

At the bottom, the unit of transfer is a logical sector of 512 bytes, or 4096 bytes on a device formatted with 4K logical sectors. Physical sectors on modern drives are 4096 bytes even when the logical size is 512. That asymmetry matters for atomicity and is taken up in Torn Writes and the RAID Write Hole.

Persistence domains are physical, and a host crash is not a device crash

A persistence domain is the set of components whose contents survive a particular failure together. The definition is relative to the failure, not absolute. If the whole rack loses power, only the media survives and every cache above it is gone. If only the host crashes while the storage array stays powered and reachable, the array’s cache survives and the host’s page cache does not. If the failure is a process kill, everything survives, because nothing outside the process address space was involved. Software that assumes one failure while defending against another gets the guarantee wrong in a way no test will reveal.

Intel’s persistent-memory platform names two relevant domains. Asynchronous DRAM refresh, ADR, uses residual platform energy to flush the memory controller’s write buffers into the DIMM when power fails; the CPU caches are not covered, so a dirty line still in L1 is lost. Extended ADR, eADR, extends the flush to the cache hierarchy, so a store that has retired is durable without an explicit cacheline flush instruction. The distinction is the difference between needing a flush and not needing one, and it is a property of the platform rather than of the program.

Durability, ordering, and atomicity need three different mechanisms

The three properties are usually discussed as one and are not. Durability says the bytes are on media after the crash. Ordering says that if write B is present after recovery then write A, issued before it, is also present; this is what lets a filesystem publish a pointer only after the block it points to exists. Atomicity says a group of writes either all appear or none do, and the media’s smallest atomic unit is one sector, so anything larger needs help.

Ordering cannot be inferred from durability. A device with a large queue may complete two writes in the opposite order from the one in which they were issued, and both can be durable after the crash even though an intermediate state existed on media in between. Ordering also cannot be inferred from the order of the write calls: the page cache, the block layer’s scheduler, and the device’s own queue all reorder. The mechanisms that impose it are a cache flush before a write and a write that bypasses the cache, described in Cache Flushes and FUA.

Atomicity for anything above one sector is built out of ordering plus a recovery procedure. A single 512-byte sector write is atomic on the media in the sense that the drive either programs it or does not; a 4 KiB write spanning eight such sectors can be interrupted after any of them, and the result is a torn write whose halves belong to different versions of the data. Logging and shadow paging are two ways of arranging writes so that the only observable torn state is a state the recovery code knows how to finish or discard.

The acknowledgement is only as honest as the device

fsync on Linux walks the file’s dirty pages through writeback, waits for the block layer to finish them, and issues a cache-flush command to the device. When it returns, the data is in the device’s persistence domain, which is media if the device honors the flush and its cache is genuinely volatile. Some devices do not honor it, some report a cache they do not have, and some are configured with write caching disabled in a way that makes the flush a no-op. In each case the call returns success and the guarantee is void. This is not a defect software can detect: the flush command has no response that distinguishes “cache emptied” from “command ignored”.

The cost is why the guarantee is not free on every write. Programming a NAND page takes tens of microseconds; a flush that must drain a queue of pending writes and wait for the media can take milliseconds, and it serializes everything behind it because the cache cannot be reported empty while writes are still in flight. A filesystem that issued one per 4 KiB write would collapse to a few hundred operations per second. Group commit exists for this reason: many transactions share one flush, and the cost per transaction falls as the batch grows.

int fd = open("/var/lib/app/state", O_WRONLY | O_CREAT | O_TRUNC, 0644);
write(fd, buf, 4096);   /* bytes live in the page cache; no media involved */
fdatasync(fd);          /* data and file size reach the device domain */
close(fd);              /* releases the descriptor; guarantees nothing more */

The sequence above leaves the file’s contents durable and its directory entry possibly not. A crash between fdatasync and the directory’s own flush can leave a file whose name does not resolve, because the name lives in a directory block that was never written. Making the creation durable takes a second synchronous operation on the directory itself.

What a crash-consistency argument has to contain

A crash-consistency argument is a claim about the set of states a reader can observe after an arbitrary power loss. It has three parts: an enumeration of the states the on-media format permits, an invariant that every such state satisfies, and a recovery procedure that turns any permitted state into a usable one. A filesystem with a journal argues that every post-crash state is either “journal not committed, old tree intact” or “journal committed, replay produces the new tree”. A filesystem with shadow paging argues that the root pointer names exactly one complete tree, whichever generation survived.

Both arguments depend on the flush and FUA placement being what the code believes it is, which is why the ordering primitives deserve their own treatment before the recovery schemes do. The two ends of that dependency are the syscall contract in What fsync Actually Waits For and the recovery procedures that assume a root is either wholly old or wholly new, as in Shadow Paging and Copy-on-Write Trees.

Where the syscall boundary stops being interesting

write returning, close returning, and even rename returning all say nothing about media. mmap with MAP_SHARED is the same story from a different angle: the store goes into a dirty page-cache page, msync with MS_SYNC is what pushes it out, and the ordering of stores inside the mapping follows the CPU’s rules rather than the file’s. O_SYNC makes each write wait for the data, and O_DSYNC waits for the data plus the metadata needed to retrieve it, a narrower and cheaper promise that omits timestamps. Neither covers the directory entry that names the file.

The practical consequence is that a program has to name the failure it survives and the state it needs after recovery, then ask for exactly those guarantees. A checkpoint that can be regenerated needs nothing. A database log needs ordering, so it needs flush and FUA placement rather than a bare write. A file whose absence would break a boot needs the directory entry as well. Anything stronger than the device’s own promise has to come from redundancy above it, because the storage stack ends at a command interface that can only be trusted as far as the firmware is.