What fsync Guarantees and What It Costs on Real Devices

Durability is the property that a returning call means the bytes survive power loss. Every database, write-ahead log, and configuration writer depends on some call providing it, and almost all of them are slower than they need to be because they assume the call is one thing. In practice fsync composes three separate guarantees — data written, metadata written, device cache flushed — and each has a different cost and a different way of failing.

The three parts of the contract

POSIX requires fsync to transfer all data and metadata for a file to the storage device and, where the device reports completion through a cache, to wait until that completion. fdatasync is the same except that metadata not required for subsequent reads of the data may be skipped. Concretely, a write that extends a file changes the file size, and that size is metadata required to find the data, so fdatasync must still flush it; a write that overwrites an existing block changes no metadata, so fdatasync may do nothing beyond flushing the data.

sync_file_range has no durability semantics at all. It starts writeback of a range and can optionally wait for the writes to be issued or completed, but it does not flush the device cache and does not order the write with respect to metadata. It exists so a program can push dirty pages toward the device early and pay the round trip later. Used as a substitute for fsync, it gives no crash guarantee.

Linux’s fsync also flushes the device’s volatile write cache, and does so by sending a flush request to the device. That flush is the expensive part. On a device that acknowledges writes into a cache without the data being on the medium, an ordinary write is cheap and the flush is what converts a batch of writes into durable state. On a device with no volatile cache, or one configured in a mode where writes are not cached, the flush has little to do but still travels the queue.

Where the round trip comes from

Take a log writer that appends a record and calls fdatasync per record. Each call dirties pages, starts writeback, waits for the device to complete the write, then issues a flush and waits for that. With nothing else in flight, the time per record is at least one device write latency plus one flush round trip. For a device whose write completion takes a fraction of a millisecond and whose flush takes a comparable time, the achievable rate is limited by the number of round trips, not by bandwidth, and the transferred size is irrelevant below the request size the device handles.

This is why group commit exists. If the state to be made durable is a batch of records, a single fsync after the batch costs one round trip for the batch, so the per-record cost divides by the batch size. The constraint is that the batch must be a unit that makes sense after a crash, which is a property of the log format rather than of the file system.

The same accounting explains why O_DSYNC and the RWF_DSYNC flag on individual writes are not automatically worse or better than explicit syncing: they make each write durable at the same per-call cost, just pushed into the write path.

Ordering constraints the filesystem adds

fsync orders a file’s data against itself, but the guarantee that matters in a crash is ordering between objects. The canonical sequence for replacing a file safely is: write the new contents to a temporary name, sync the temporary file, rename it over the target, then sync the directory. The first sync ensures the new data is durable before the name exists; the rename is atomic in the directory so a reader sees old or new, never a mixture; the second sync makes the new name durable, so a crash after the rename cannot revert to the old file while the new data is on the medium.

Omitting the directory sync is the most common defect in this pattern. Directory updates are metadata and are journaled, but journal commit does not immediately put a directory entry on the medium, so a crash can leave the rename lost with the data present — or, worse, with neither. Directory writes take the same buffered path as file writes, described in What write() Actually Does Before Data Leaves the Process; the difference is only that the object being dirtied is the parent directory rather than the file.

The converse error also occurs: syncing the directory after every file creation in a loop, which serialises creation against a flush round trip each time and can dominate the cost of writing a directory of small files.

Failure is not a single errno

fsync can fail after reporting success on earlier calls. A device that loses a write in its cache, or a filesystem that encounters an unrecoverable metadata problem, may mark the file’s state as invalid and fail subsequent syncs. The data from the earlier writes is then not reliably present even though those writes returned zero.

POSIX leaves the behaviour after an fsync error unspecified with respect to future operations, and the practical rule from systems that implement write-ahead logs is that once an fsync fails, the file is in an unknown state and only a retry followed by, when that also fails, treating the file as lost is safe. A retry is required by the interface on interruption; the kernel may have completed the work and still returned an error for a signal, so a single failed call is not proof of data loss, but a repeated failure is.

There is also an error path that is not the caller’s: on some storage, a write can be dropped and reported only on a later flush. The flush is where the device reports it, so the failure appears on a call that wrote nothing new. An application that only checks the return value of the write it just performed misses it.

Cost shapes on modern storage and where they mislead

A consumer solid-state device has a volatile cache that it flushes on command, a small capacitor or none, and firmware that may reorder aggressively. Its flush cost is not the same as that of an enterprise device with a power-loss-protected cache, where a flush is nearly free because the cache is guaranteed to be written. The same code on the two devices can differ by an order of magnitude per fsync, and neither figure generalises to a network volume, where a flush becomes a round trip to another machine.

Caching layers above the filesystem change the arithmetic again. A write-back cache in a hypervisor acknowledges a flush only after it has made the data durable somewhere, so the guest’s fsync inherits the host’s flush cost plus the interception cost. A network filesystem may batch flushes from many clients into one, which makes one client’s fsync cheap while it is not alone and expensive when it is.

For all these reasons, a measurement of fsync throughput should state the device class, whether the file system was mounted with barriers enabled, and whether the test issued one sync at a time or many concurrently. A number without that context measures a configuration, not a device. The queue-level view of how those writes were merged before reaching the device is covered in Writeback Scheduling and Merging at the Block Layer.

Alternatives and what they give up

Batting groups of records behind one fsync is the main practical lever. A second lever is to accept bounded loss: sync periodically and on shutdown, so a crash loses at most the interval. A third is to avoid the round trip on the device entirely by writing to a device that guarantees its own cache, which costs money rather than latency.

Direct I/O is orthogonal. It removes the page cache and therefore removes the ambiguity about whether a write hit cache, but it still requires a flush for durability and still pays a round trip per sync. What it changes is that the data does not need to be dirtied and later flushed by the flusher, which matters for the rate of writes rather than for the durability handshake.

Finally, the guarantee is scoped to one file and, for the directory, one directory. Making a multi-file update durable requires syncing all of them in an order that makes each intermediate state recoverable, or writing the update as a single log record that is applied on recovery. The second is what journals are for, and it is the only structure that makes a crash in the middle of a multi-object update cheap to handle.