Durability Barriers, Flushes, and What fsync Actually Waits For

Every guarantee in this cluster bottoms out in a primitive that forces data onto media before returning. Journaling’s commit record, copy-on-write’s root pointer, and ordered mode’s data-before-metadata dependency are each a claim that some write preceded another, and each is only as strong as the storage stack’s willingness to honour that claim. The stack between the filesystem and the platter or the flash cells has several caches, and every one of them can reorder or lose a write that the layer above believed durable.

The path a flush takes

An fsync on a file descriptor does not directly command the device. It asks the filesystem to make the file’s data and the metadata needed to find it durable. The filesystem writes its dirty pages, then submits a flush request to the block layer, which translates it into a cache-flush command for the device: SYNCHRONIZE CACHE in the SCSI command set, FLUSH CACHE in the ATA set, a corresponding command in NVMe. The device is supposed to write its volatile cache to media and report completion. Only when that report arrives does the flush return, and only then does the filesystem’s commit sequence advance.

Each layer in that path is a place where the guarantee can weaken. A host bus adapter with a write-back cache, a RAID controller with a battery-backed cache, a hypervisor presenting a virtual disk backed by a file, a network block device whose server acknowledges before persisting: all of them sit between the filesystem and the media, and all of them must propagate the flush rather than absorb it. The device driver’s job is to pass it through; the firmware’s job is to implement it. When a drive’s cache is configured to ignore flush commands for performance, the command returns success without doing anything, and the filesystem has no way to tell.

This is not a hypothetical class of failure. Drives with volatile write caches that report flush completion immediately have been documented, and virtualized storage that ignores flushes is a common misconfiguration because it makes benchmarks look better. The consequence is precise: every journaling guarantee becomes a lie, because the commit record can reach media before the payload despite the barrier. The checksum over the transaction detects the resulting damage, but detection is not prevention; the filesystem refuses to replay a corrupt transaction and the operation is lost rather than half-applied, which is the best available outcome.

What fsync must cover, and what it does not

POSIX requires that fsync return only after the file’s data and the metadata required to retrieve it have been transferred to the storage device in a way that survives a system crash. The wording matters: the guarantee is about the file, not the filesystem, and it is about the metadata needed to find the file’s data, not all metadata.

The classic gap is the parent directory. Creating a file writes an inode and adds a directory entry. Calling fsync on the new file’s descriptor makes the inode and its data durable; it does not necessarily make the directory entry durable, because the directory is a different inode and the filesystem has no obligation to flush it. A crash after the file’s fsync returns can leave the file’s blocks allocated and the inode populated but the name absent from the directory. The data is there and unreachable. To make the name durable, the caller must also open the directory and fsync it, a requirement that is documented in the filesystem’s own semantics and that most applications get wrong by omitting the second call.

The reverse gap is subtler. An fsync on a directory does not necessarily flush the contents of the files the directory names; it flushes the directory’s own metadata, which includes the entries. So a sequence that creates a file, writes to it, syncs the file, and syncs the directory produces the full guarantee, while syncing only the directory produces the name but not the contents.

fdatasync narrows the scope further: it flushes the file’s data and the metadata strictly required to read that data, namely size, block pointers, and timestamps where the filesystem’s semantics require them, but not metadata that only affects later reads of attributes. For a file whose size did not change, it can avoid an extra metadata write. For a file that grew, the size change is exactly the metadata needed to read the new data, so the saving disappears.

The ordering fsync imposes inside the filesystem

An fsync is not a single device command. The filesystem must ensure that, in addition to the file’s own blocks, any journal transaction whose metadata the file depends on has committed. In ordered mode this means the data blocks must be written before the transaction’s commit record, which the filesystem achieves by forcing the data and then waiting for the current transaction to commit. A single fsync can therefore trigger a transaction commit that covers unrelated operations batched into the same group, which is why the latency of a sync is not a function of the file’s size alone.

The wait is the mechanism behind the well-known behaviour that fsync latency under concurrent load is much worse than fsync latency in isolation. A transaction group commits for all its members at once, and the commit is serialized: the barrier must complete before the commit record is written, and the next group cannot commit until the previous one’s commit record is durable. Every syncing caller in the group waits for the slowest write in it. Reducing the group size reduces the wait but increases the number of barriers, so the optimum depends on the ratio of sync latency to device flush latency, a ratio that differs by orders of magnitude between a battery-backed RAID controller and a consumer flash drive.

The filesystem also has to handle the case where the data it must flush is not contiguous or not present. A file whose pages were written through mmap and never dirtied by a write call has its dirty pages in the page cache with no filesystem-visible record of which transaction they belong to; the fsync must walk the file’s page range and write them, which for a large mapping is proportional to the mapped size. This is why msync exists as a separate call and why relying on fsync alone for mapped data is a mistake.

Copy-on-write’s use of the same primitive

A copy-on-write filesystem needs the barrier in a different place. Its commit sequence writes the new tree, flushes, then writes the root. The flush covers all the new blocks; the root write is the atomic publication. If the root write itself is not atomic, because the device tears a sector write, the design needs redundant roots with generation numbers and checksums, as discussed in the shadowing article.

The performance consequence is that a copy-on-write commit costs a full flush of every block written in the transaction group, with no possibility of a partial commit. A journaling filesystem can commit a transaction whose payload is only a few blocks and let other blocks wait; a copy-on-write filesystem’s transaction group commits as a unit, and its size is bounded by the flush latency the workload can tolerate. This is why copy-on-write filesystems expose a commit interval as a tunable and why synchronous writes to a copy-on-write volume can be slower than the same writes to a journaled in-place filesystem: the former pays a group-wide flush, the latter a per-transaction one that can be smaller.

Neither design escapes the underlying constraint: a flush is a device operation whose latency is set by the media and the cache, and no amount of filesystem cleverness reduces it. The only lever is batching, which trades durability granularity for throughput. A filesystem that commits every 30 seconds has lower write amplification from barriers than one that commits every second, and loses up to 30 seconds of committed-looking work on a crash.

What a crash test can and cannot establish

Because the guarantee depends on hardware behaviour that the filesystem cannot observe, verifying it requires either a device whose cache semantics are known or an external power-cut mechanism that actually removes power at a controlled instant. Software that simulates a crash by killing a process tests nothing about durability; software that unmounts cleanly tests nothing about recovery. The meaningful test is to write a known sequence, cut power at a chosen point in the sequence, and check what the filesystem presents after mount against the guarantee it claims.

That kind of test is what distinguishes a filesystem’s claimed semantics from its actual ones, and it is the reason the ordered, journalled, and writeback modes are documented as distinct promises rather than as performance settings. The mode selects which writes the filesystem orders around the barrier; the barrier is what makes the order real. A stack where the barrier is absorbed by a cache produces writeback behaviour regardless of the mode selected, and no amount of filesystem configuration recovers the lost guarantee. The dependency-ordering alternative has the same exposure: its whole correctness argument is a set of ordering constraints, and those constraints are only as good as the flush that enforces them.