What fsync Actually Waits For, and Why the Directory Counts

A program writes a configuration file to a temporary name, calls fsync on the descriptor, renames the temporary over the real name, and returns. Every step succeeded. A power loss follows, and after recovery the real name is missing entirely: the old file is gone, the new file has no directory entry, and the temporary name may or may not resolve. Nothing in the sequence was wrong except the assumption that flushing the file’s contents also flushed the change to the directory that names it. They live in different objects and are written back independently.

Data, size, and metadata are three separate promises

POSIX defines fsync as scheduling the file’s in-core data and metadata for transfer to the storage device and waiting for completion. It defines fdatasync more narrowly: it waits for the data needed to retrieve the file’s contents, and may skip metadata that does not affect that retrieval. The distinction is not cosmetic. A file’s modification timestamp lives in an inode; the inode is a different block from the data blocks; a filesystem may keep the timestamp dirty and write it much later without any risk to the bytes a reader would see. Under fdatasync that is permitted, and under fsync it is not.

The size is the awkward case. Appending to a file changes the length recorded in the inode, and a reader that trusts the length will see the appended bytes only if the inode’s new length is durable. So fdatasync must flush the inode when the operation changes the file’s length, which means appends and truncations get the same treatment as a full fsync, while an in-place overwrite of existing bytes does not. Filesystems exploit this: an overwrite of a block already allocated needs only that block flushed, and the inode can stay dirty.

None of this touches the directory. The directory is a file whose contents are a mapping from names to inode numbers, and the inode number of a newly created file is recorded there by a write to a directory block. fsync on the file’s descriptor cannot reach it, because the descriptor refers to the file, not to the directory entry that names it.

Why a flushed file can still be nameless

Consider open("/tmp/x", O_CREAT) followed by fsync. The inode now exists on media with a nonzero link count, and the data is durable. The directory block containing the entry x is a dirty page belonging to the directory’s inode, and it is scheduled by the writeback machinery like any other dirty page. If the crash arrives before that page reaches media, recovery sees an inode with a link count of one and no name pointing to it. Some filesystems run a consistency pass at mount that reclaims such orphans; some leave the space allocated and unreachable. Either way the name is gone.

The correct protocol adds one more synchronous operation on the directory:

int dfd = open("/etc", O_RDONLY | O_DIRECTORY);
int fd  = open("/etc/app.conf.tmp", O_WRONLY | O_CREAT | O_TRUNC, 0644);
write(fd, buf, len);
fsync(fd);            /* contents and inode durable */
close(fd);
rename("/etc/app.conf.tmp", "/etc/app.conf");
fsync(dfd);           /* the directory entry change durable */
close(dfd);

Order matters in both directions. The fsync on the file must precede the rename, because the rename publishes a name that may point to an inode whose blocks are not yet on media; a crash between them would leave a resolvable name for a file full of stale or zeroed blocks, which is worse than a missing name. The fsync on the directory must follow the rename for the reason above. Neither operation implies the other.

rename itself is atomic with respect to other processes in the sense that a concurrent reader sees either the old name or the new one, never a partial state. That atomicity is a property of the running kernel’s namespace operations, and it does not extend across a crash unless both the old directory’s and the new directory’s blocks reach media. When the source and destination are in different directories, two directory blocks are involved and both need flushing; when they are in different filesystems, rename fails with EXDEV and the caller must copy instead, at which point there is no atomic publication at all and a temporary name plus a link or a marker file has to carry the state.

The guarantees stop at the device boundary

fsync returning means the kernel has finished its part. The writeback it triggered completed, the block layer’s queue drained, and a cache-flush command was issued to the device. What the device did with that command is outside the kernel’s knowledge. A drive with a volatile write cache that ignores the flush will report success and lose the data. This is why the guarantee is often stated as “durable to the device’s persistence domain”, a phrase that quietly relocates the problem to firmware. The domain is relative to the failure being survived, as set out in Persistence Domains and Write Ordering, and a host crash and a device crash leave different sets of components intact.

The same reasoning explains why fsync is expensive and why its cost is lumpy. On a device that must program NAND, the flush waits for the media, and the wait is on the order of the device’s program time rather than the transfer time. On a device that acknowledges into a battery-backed cache, the wait is much shorter but the durability now depends on the battery. A filesystem that batches many dirty inodes into one flush amortizes the cost across them, which is why syncfs exists: it flushes everything on one filesystem in a single pass rather than one file at a time. The command that the flush becomes, and the write that carries its own barrier, are described in Cache Flushes and FUA.

syncfs is not a substitute for fsync in a protocol. It gives no ordering between the objects it flushes, so it cannot establish that the file’s contents were durable before the rename. It is useful at a coarse boundary, such as before suspending a machine or before taking an external snapshot, where the requirement is “nothing dirty remains” rather than “this particular edge is ordered”.

Protocols that survive the crash

A protocol is crash-safe when every reachable post-crash state is one the reader can interpret. Three patterns cover most cases.

The first is write-then-rename, shown above. Its strength is that the published name always refers to a complete file, because the file was flushed before the name existed. Its weakness is that the old version is gone the moment the rename lands, so there is no window in which both versions are readable; a reader that needs the previous contents has to keep a separate copy under another name.

The second is the two-file marker. Write the new content under a temporary name, flush it, then create a small marker file that names the temporary file, flush the marker, then rename. Recovery reads the marker and completes the rename if it exists. This buys the ability to detect an interrupted publication, at the cost of an extra flush and an extra inode.

The third is a log with a commit record, which is the subject of Write-Ahead Logging and the Commit Record. Its advantage is that one flush covers an arbitrary number of changes, so the per-operation cost of durability falls as the transaction grows. Its cost is that recovery must replay, and replay must be idempotent.

Where these calls mislead

fsync on a directory descriptor is not portable in the sense programmers expect. POSIX does not require that a directory can be opened for reading on all systems, and some platforms return an error for fsync on a directory. Code that assumes it works everywhere will fail in a way that is easy to miss if the return value is ignored. Ignoring the return value is itself a recurring defect: fsync can fail with ENOSPC after the write succeeded, because the metadata the flush needs to allocate has nowhere to go, and the data is then lost while the program believes it was saved.

There is also a subtlety about what a descriptor refers to after a rename. An open descriptor keeps referring to the same inode, not to the name, so an fsync after the rename flushes the file that was renamed, which is the intended object. But a program that reopens by name after the rename may be holding a descriptor to a different inode if the rename replaced a file that another process had open. Neither descriptor is wrong; they refer to different objects, and the durability question has to be asked per object.

Finally, the page cache is shared. A write by one process dirties a page that another process’s read will see immediately, before any flush. The kernel’s consistency is not in question; the confusion is about whether visibility implies durability. It does not, and no amount of reading back what was written establishes anything about media. The only evidence of durability available to a program is the return of a synchronous call, and that evidence is conditional on the device honoring its side.

The layer below this contract, the way a flush and a forced unit access are expressed to the device, belongs to the block layer rather than to the syscall interface. Above it sits the question of what a program should do with the guarantee once it has one, which is where logging and checkpointing begin.