Cache Flushes and FUA: The Two Ordering Primitives Below the Filesystem

A journaling filesystem writes a commit block and then writes a data block that the commit block points to. The code issues the writes in that order and expects them to reach media in that order. They do not: the block layer holds both in its queue, the device holds both in its cache, and the drive is free to program whichever page it likes first. If the data block is the one that lands after a crash and the commit block is not, recovery reads a commit record that claims a block which does not exist. The fix is not a different ordering of write calls; it is a pair of commands that turn the queue’s freedom into a constraint.

The device cache is a separate persistence domain

A drive that reports completion into a write-back cache has told the host that the bytes are in the cache, not that they are in the media. The distinction matters because the cache is volatile: it is backed by a small amount of DRAM, kept alive by the same power that keeps the host alive. On a sudden loss, its contents are gone. On a clean shutdown, the drive drains it. Between those extremes is every crash a program has to survive.

The cache exists for performance and its size is the point. A drive can accept a large burst of writes at link speed because it is only copying into DRAM, then program the media in the background at a much lower rate. NAND page programs take on the order of tens of microseconds each, and the interface can transfer a page in far less, so the cache absorbs the difference. A host that wants the media to be current has to ask, and asking is the whole of the durability mechanism.

Two commands do the asking. A cache flush tells the device to write everything currently in its volatile cache to media and report completion only when that is done. A forced unit access tells the device to write this particular unit to media before reporting completion, bypassing the cache for that operation. The names vary by transport, SYNCHRONIZE CACHE and WRITE FUA in SCSI, FLUSH CACHE and a write with the FUA bit set in ATA, but the semantics are the same pair, and the block layer exposes them as two request flags.

A flush is a barrier, not a write

A flush has no payload. It is a request that the queue drains and the device empties its cache, and its completion carries no data. Because it is a barrier, its cost is proportional to the queue depth in front of it: every write already accepted has to reach media before the flush can report. A device with 32 commands in flight and a program time of tens of microseconds per page has a flush latency in the high hundreds of microseconds at minimum, and the media’s own scheduling can make it worse.

That cost structure explains why flushes are batched. If a transaction needs one flush at commit, then N concurrent transactions that commit together need one flush, not N. The group commit machinery in a database or a filesystem collects the transactions that arrive while a flush is in flight and issues a single flush covering all of them. The per-transaction cost then falls with the batch size, and the latency of an individual transaction rises with the queueing delay. This is the fundamental trade of durable storage: latency for amortization.

A flush also has a failure mode that matters for correctness. If the device does not implement it, the command returns success anyway, because there is no defined way to report that it was ignored. Some drives are configured to treat the cache as write-through and then implement the flush as a no-op; that is correct behaviour for the configuration, and the durability guarantee is genuinely satisfied, since nothing was cached. Others simply lie. Software cannot distinguish the two cases, which is why the honest statement of a filesystem’s guarantee is conditional on the device honoring the command.

FUA is a flush fused into a write

A forced unit access carries the data and the barrier together. The device must write that unit to media before it reports completion, so the completion itself is the barrier for that write. It is cheaper than a separate flush when the number of writes being ordered is small, because it does not have to drain unrelated requests that are already in the queue.

The catch is that FUA applies to one write, and ordering needs a relationship between two. To make write A precede write B, the sequence is: write A with FUA, wait for its completion, then write B normally. B may sit in the cache afterwards, but A is on media and the crash cannot reorder them, because A’s presence was established before B was even issued. The reverse arrangement, FUA on B, establishes nothing about A.

A common arrangement uses FUA for the commit record and a flush for the surrounding data. The data blocks are written normally and then a flush drains them; the commit record is written with FUA so that its arrival implies the flush’s completion. That saves a separate flush command per commit, at the cost of making the commit record’s write synchronous, which serializes the commit against the media. Which arrangement wins depends on the ratio of data to commit writes and on whether the device implements FUA efficiently; some devices implement FUA by internally issuing a full flush, which makes it no cheaper than the separate command. The commit record itself, and the validation that makes a torn commit detectable, are covered in Write-Ahead Logging and the Commit Record.

Building a barrier in the block layer

Linux models these as request flags on a bio or a request: the flush flag and the FUA flag. A filesystem issues a flush by submitting a request with the flush flag and waiting, or by using the ordering helpers that submit a flush and a following write as a pair. The details are in the kernel’s block layer, and the important property is that the flags survive down to the transport: a flush flag becomes SYNCHRONIZE CACHE, a FUA flag becomes a write with the FUA bit or a WRITE FUA command.

#include <linux/blkdev.h>

/* Data first, then a cache drain, then a commit record that carries its own
 * barrier. blkdev_issue_flush() becomes SYNCHRONIZE CACHE on the transport. */
static int commit_sequence(struct block_device *bdev, struct bio *data,
                           struct bio *commit)
{
    submit_bio(data);                    /* may remain in the device cache */
    if (blkdev_issue_flush(bdev) < 0)
        return -EIO;                     /* cache could not be drained */
    commit->bi_opf |= REQ_FUA;           /* becomes WRITE FUA */
    if (submit_bio_wait(commit) != BLK_STS_OK)
        return -EIO;
    return 0;                            /* commit record is on media */
}

Two implementation details are easy to get wrong. The first is that a flush issued before the writes it is supposed to order is useless; the barrier has to follow the writes. The second is that the ordering must hold across the entire stack, including any device-mapper layers in between. A software RAID or an encrypted block device that reorders or splits requests can break an ordering assumption made above it unless it propagates the flags, and a layer that silently drops FUA forces the caller to use an explicit flush instead.

Where the abstraction leaks

The barrier primitives are defined relative to a device, and a filesystem built on several devices has to reason about each one. A logical volume striped across four drives has four caches; a flush to the volume has to reach all four, and a crash can leave them inconsistent with each other if the flush only reached some. Redundant arrays handle this by writing to all members and relying on parity or mirroring to repair a member that is behind, which is why the write hole exists and why it is discussed in Torn Writes and the RAID Write Hole.

Virtualized storage adds another layer with the same property. A hypervisor’s virtual disk backed by a file on a host filesystem has to translate the guest’s flush into a host fsync, and a hypervisor that acknowledges the guest’s flush without performing the host’s has broken the guarantee in a way the guest cannot detect. This is why the flush semantics of a virtual disk are part of its contract, and why a guest that assumes they hold is trusting two stacks rather than one.

There is also the question of what a flush does to unrelated data. The command drains the whole cache, so writes from other files, other filesystems on the same device, and other processes all become durable along with the ones the caller cared about. That is harmless for correctness and occasionally useful, but it means a flush’s latency is set by the busiest writer on the device, not by the caller’s own queue. A program that issues many small synchronous writes on a shared device pays for its neighbours’ backlog.

The cost is in the wait, and the wait is unavoidable

Every durable protocol on a device with a volatile cache needs at least one round trip to the device after the last write it cares about. That round trip is the floor on commit latency, and no amount of batching removes it, only amortizes it. A protocol that appears to avoid the round trip is either writing to a device whose cache is non-volatile, which is a different persistence domain, or it is not actually durable.

The practical consequence for design is that the number of barriers per unit of work is the quantity to minimize. One barrier per transaction is the baseline; one barrier per batch of transactions is better; one barrier per checkpoint that covers thousands of operations is better still, at the cost of losing the work since the last checkpoint. That trade is the subject of checkpointing and of logging, and both rest on the two primitives described here.