Cache Flush, FUA, and What a Write Completion Means

A write that the device has acknowledged is not necessarily a write that survives a power failure. The acknowledgement means the device has accepted the data, and a device with a volatile write cache accepts data into DRAM and reports completion before programming it to NAND. The cache makes writes fast because the host is not waiting for a program operation, and it makes the durability of an acknowledged write conditional on the device having enough time and power to destage. The ordering discipline that journals depend on is built on top of the operations that make the condition explicit.

Three levels of acknowledgement

The block layer’s contract distinguishes three outcomes, and the difference between them is the entire subject.

A plain write completes when the device has taken responsibility for the data in a way that survives a power loss, or when it has taken the data into a cache and the host is not asking for more. Without a cache, or with the cache disabled, the completion means the data is in non-volatile media. With a cache enabled, it means the data is in the cache, and a power loss discards it.

A force-unit-access write, FUA, carries a flag asking the device to write the data through the cache to media before reporting completion. The write itself is the unit of durability: when it completes, that specific write is on media. The flag exists because the alternative, a separate flush command, costs a round trip for each group of writes, while FUA can be attached to the writes that need it. What FUA can make durable is bounded by the device’s atomic unit, which is discussed in logical block size and alignment: a write larger than that unit may itself be split on media.

A flush command asks the device to write everything currently in its cache to media and to report completion when that is done. The flush is not tied to a specific write; it is a barrier over everything the device has accepted so far.

The three correspond to different command encodings. In the SCSI command set, the flush is SYNCHRONIZE CACHE, and the FUA bit is part of the WRITE command’s CDB. In the ATA command set, the flush is FLUSH CACHE or FLUSH CACHE EXT, and FUA is a bit in the WRITE DMA FUA EXT command, supported only on some devices. In NVMe, the flush is a Flush command applied to a namespace, and FUA is a bit in the Write command. The block layer models both with a queue flag for whether the device has a volatile cache and a request flag for whether the request requires FUA.

Why a flush is a barrier and a write is not

The cache reorders. A device with a write-back cache is free to destage its contents in any order, and it is free to report completion of writes in any order, subject only to writes to the same logical block. Two writes to different blocks, issued one after the other, may reach media in either order. The filesystem’s journal depends on the opposite: the payload must be on media before the commit record, or a power failure leaves a commit that names payloads that were never written.

A flush between the payload writes and the commit write creates the order. The sequence is: write payloads, issue flush, wait for the flush to complete, write commit, issue flush again if the commit itself must be durable. The first flush guarantees the payloads are on media before the commit write is even submitted, and the device’s reordering cannot violate a dependency that has already been resolved. The second flush makes the commit durable, and its completion is the point at which the transaction is committed.

The cost is two round trips and two device cache drains per transaction. The first flush waits for the device to destage everything pending, which on a device with a large cache can be many programs; the second does the same for the commit block alone. A filesystem that commits frequently pays this per commit, and the latency is why the commit interval is a tunable.

FUA offers a cheaper shape. If the commit write carries FUA, the device writes it through the cache and reports completion only when it is on media. The payloads need no flush if the device’s ordering rules ensure that a later write cannot reach media before an earlier one to the same… but that is exactly what is not guaranteed across different blocks. The correct use of FUA is to attach it to the commit write and still flush before it, or to attach it to every write whose ordering matters, which reduces to per-write durability.

#define PAYLOAD_OFF 0
#define COMMIT_OFF  4096
#define COMMIT_LEN  512

/* The barrier sequence, written against the raw device. */
static int commit(int fd, const void *payload, size_t n, const void *commit_rec)
{
    if (pwrite(fd, payload, n, PAYLOAD_OFF) != (ssize_t)n)
        return -1;

    /* First barrier: the payload must be on media before the commit exists. */
    if (fsync(fd) != 0)
        return -1;

    if (pwrite(fd, commit_rec, COMMIT_LEN, COMMIT_OFF) != COMMIT_LEN)
        return -1;

    /* Second barrier: the commit record itself must be durable. */
    return fsync(fd) == 0 ? 0 : -1;
}

On a device that reports no volatile write cache, both fsync calls return without waiting, because every write completion is already durable. On a device with a cache, each is a full drain of what the device holds, which is why the sequence is expensive in proportion to how much unrelated write traffic is in flight when the barrier is issued.

The REQ_FUA flag carries the same requirement in the block layer’s own vocabulary; the layer translates it into whatever the device supports. A device that has no FUA support and reports no volatile cache gets a plain write, which is correct because its completions already mean durable. A device that has a volatile cache but no FUA support forces the block layer to emulate FUA by issuing a write followed by a flush, which is the same cost as the explicit barrier.

What the device may still do

A flush is a promise by the device, and the promise is only as good as the firmware. Documented failure modes exist at every level.

The first is a device that reports a flush complete before the data is on media. This can be a firmware bug, or a consequence of the device’s power-loss protection being absent: a drive without capacitors cannot destage its entire cache on power loss, so a flush that returns before the cache is empty is a lie. Enterprise drives with power-loss protection have capacitors sized to flush the cache, and the flush’s completion is meaningful because the hardware can finish the work without host power.

The second is a device that ignores the FUA bit and treats the write as a plain cached write. The bit is advisory in the sense that a device which does not support it may report that it does not, but a device that accepts the command and drops the flag is not detectable from the host: a commit record whose payloads are missing looks exactly like one whose payloads were never written. The only defence available above such a device is a checksum over the payload, computed by whoever wrote it, so that a replayed transaction whose data is incomplete is rejected rather than applied.

The third is a virtualized or layered device in between. A host’s flush may be satisfied by a hypervisor’s page cache, which then writes to the backing store later. The guest sees a completed flush; the host has not destaged anything. The same applies to a software RAID layer, a device mapper target, or a network block device whose server acknowledges writes before committing them. Each layer must propagate the flush, and each layer that does not turns the guest’s durability guarantee into a cache hint.

The block layer tracks the property in the queue’s flags: whether the device has a volatile write cache, whether it supports FUA, and whether it supports discard. The filesystem reads those flags and chooses its commit strategy accordingly. A filesystem on a device with no cache and no FUA can skip the flush entirely, because completions are already durable; that is the configuration where write ordering is cheapest.

Ordering without a flush

The block layer can also enforce ordering in software, by holding requests in the queue until earlier ones complete. This is what an ordered request or a barrier request does when the device lacks the hardware feature. The cost is the same as a flush in latency, because the queue must drain, but the mechanism is the host’s rather than the device’s and does not depend on the device honoring a flag. Software ordering is bounded by the same per-range rule that governs discard: a request that frees a range is ordered only against writes to that range, as described in discard and TRIM semantics.

Software ordering is what the filesystem’s own write submission can rely on when it submits writes in a chain and waits for each. A synchronous write that waits for completion before submitting the next creates the order by construction, at the cost of serializing the device. The reason journals use flushes rather than serialization is that the payload writes can proceed in parallel and only the boundary needs a barrier.

The durability of the directory entry

A file’s data being on media is not the same as the file existing after a crash. The directory entry that names the file is metadata in a different block, and its creation must be durable before the application is told the file exists. The same barrier discipline applies, and the ordering is between the inode, the directory entry, and any data the file’s creation implies. A filesystem that writes the directory entry first and the inode later can, after a crash, have a name pointing at an unallocated inode; a filesystem that writes the inode first and the directory entry later can have an allocated inode with no name, which is a leak but not corruption.

The fsync interface is where the application expresses the requirement. A call to fsync must, on return, have the file’s data and the metadata needed to find it on media. The filesystem translates that into writes, a flush, and possibly a second flush, and the device’s cache behavior determines the cost. The guarantee is about the file, not about the device, and it holds only if every layer between the application and the media propagates the flush. The layers below the block device, and the devices that lie about the flush, are the reason the guarantee is stated as a contract with the storage stack rather than with the hardware.