fsync, fdatasync, and Where the Durability Boundary Actually Sits

An engine that calls write and returns success has accomplished nothing durable. The bytes are in the kernel’s page cache; they will reach the device at some point chosen by the writeback machinery, and a power failure before that point loses them. Everything an engine says about durability is a statement about what it does after write, and the details of that “after” are where the guarantees are actually made or quietly lost.

The three levels a write can reach

A write to a file passes through at least three distinct resting places, and each requires a different action to be safe.

The first is the process’s own buffer, which is under the engine’s control. Copying into the kernel ends the engine’s responsibility for the bytes but not its responsibility for their fate.

The second is the kernel page cache. Here the data is visible to every subsequent read, is coherent across processes, and is lost on power failure or kernel panic. A write returning means only this level. On a clean shutdown the kernel flushes the cache, which is why a database that never calls fsync still appears correct after an orderly reboot and fails only after a crash.

The third is the storage medium, or a cache that the device promises to preserve across power loss. Reaching it requires an explicit flush: fsync, fdatasync, or a write issued with a synchronous flag.

The gap between level two and level three is the durability window. Its width is set by the kernel’s dirty page writeback parameters, and on a busy system it is measured in seconds, not milliseconds. A database that acknowledges a commit without a flush has a durability window of that width, and the correct description of its guarantee is “committed transactions survive process crashes and clean shutdowns, but not power loss.”

fsync versus fdatasync

fsync(fd) flushes the file’s data and the metadata needed to retrieve it: size, timestamps, and the block mapping. fdatasync(fd) flushes the data and only the metadata required for a subsequent read to return the same bytes, which in practice means it may skip an mtime update.

The distinction matters when the file is being appended to and its size changes, because the size is exactly the metadata a read depends on. Appending to a log and calling fdatasync is safe only if the size update has already been committed by a previous flush or if the file system allocates the space in advance. Engines handle this by preallocating log segments and writing within the allocated extent, so that appending does not change the file’s size and fdatasync is genuinely sufficient. An engine that appends to a growing file and calls fdatasync is relying on a property the file system does not guarantee.

The performance difference is real but workload-dependent: on a file whose metadata is already up to date, fdatasync and fsync issue the same cache-flush command, and the saving is only the journal transaction for the metadata. On a file system with a journalling metadata path, skipping that transaction removes a second write and a second flush.

The flush is a device command, not a memory operation

A common misreading is that fsync waits for the kernel to finish writing. It does not, in the sense that matters. fsync must ensure the data is on a medium that survives power loss, which for a device with a volatile write cache requires issuing a cache-flush command and waiting for its completion. If the device lies about the flush — reports completion while data is still in its volatile buffer — the guarantee is broken regardless of what the kernel does.

This is why storage configuration is part of the durability story rather than an operational detail. A RAID controller with a battery-backed write cache can acknowledge a flush immediately and preserve the data across power loss, which is a genuine improvement in latency. A controller with a volatile cache that ignores flush commands converts every fsync into a no-op with a success return code, and no software setting detects it.

The file system also participates. A file system that reorders writes for locality must still honour a flush by ensuring all writes issued before it are durable before it completes. Most do; the failure mode to be aware of is a file system that batches flushes across files, which is correct for fsync on one file only if the batch’s completion implies that file’s data is durable.

/* The minimum a commit path must do, with the ordering made explicit. */
int log_append_and_sync(int fd, const void *buf, size_t len, off_t off) {
    if (pwrite(fd, buf, len, off) != (ssize_t)len)
        return -1;
    /* fdatasync suffices only because the segment is preallocated:
       no size change, so no metadata a read depends on. */
    if (fdatasync(fd) != 0)
        return -1;                 /* an error here means the data may be lost */
    return 0;
}

The error path deserves attention. A failed fsync means the kernel could not determine the state of the data: some pages may be durable, some may not, and retrying the call is not guaranteed to converge, because many kernels mark the pages clean after a failed writeback attempt and a second fsync returns success without doing anything. The defensible response to a failed flush is to treat the file as poisoned and fail the operation loudly, which is why some engines escalate a write error to a shutdown rather than retrying.

Direct I/O and what it does not fix

O_DIRECT bypasses the page cache: reads and writes go between the caller’s buffer and the device, with no copy into kernel memory. Its purposes are to avoid double buffering and to give the engine control over when I/O happens.

Double buffering is the concrete waste. A database with a 16 GiB buffer pool running on a machine with 32 GiB of RAM and a file system that caches reads will have the same pages in both caches, and the kernel’s copy is pure overhead — it duplicates memory, and it competes for capacity with the pool that the engine is actually managing. The kernel also cannot know which pages the engine considers hot, so its replacement decisions are made with less information.

O_DIRECT has strict alignment requirements: the buffer address, the file offset, and the length must each be a multiple of the device’s logical block size, commonly 512 bytes or 4 KiB. A buffer allocated with malloc is generally not aligned, so engines allocate page-aligned memory — posix_memalign or a mapped region — and size their I/O in whole blocks. A misaligned request fails with EINVAL rather than being silently handled.

The crucial point is that O_DIRECT does not by itself provide durability. It removes the page cache from the path, so a completed write has reached the device, but a device with a volatile cache may still hold it. A flush is still required for the force-at-commit boundary. O_SYNC combines the two: each write is synchronous and does not return until the data is durable, at the cost of one flush per write and no opportunity to batch. Engines that want batching use O_DIRECT plus explicit fdatasync at commit boundaries.

O_DIRECT also disables readahead, because the kernel no longer sees the request pattern. A sequential scan through a direct-I/O file therefore issues one read per page with no prefetch, which is why engines that use it implement their own readahead by issuing asynchronous reads several blocks ahead.

Sizing the I/O to the device

The block size the engine uses and the block size the device prefers are not the same number. A device may report a 512-byte logical sector and a 4 KiB or larger physical erase unit; writing a 512-byte block that lands alone in a 4 KiB unit forces a read-modify-write of the whole unit. This is why engines align pages to at least the physical block size and why a page size smaller than the device’s indirection unit produces write amplification at the device level, an effect traced in write amplification across the storage stack.

The same arithmetic applies to the log. A log write of a few hundred bytes still costs a full device transaction; batching commits into one flush amortizes it, which is the group commit mechanism described in write-ahead log record ordering. The number of flushes per second is the hard limit on commit throughput for a synchronous engine, and it is a property of the device, not of the engine’s code.

What survives what

The practical summary is a table of failure modes rather than a single guarantee. A process crash loses nothing that reached the page cache, because the cache belongs to the kernel. A kernel panic loses everything in the page cache that has not been written back. A power failure loses both the page cache and any device cache that is not protected, and a torn write during the transfer is possible for any page-sized write that was in flight. A device failure loses whatever was on that device, which is why replication exists.

An engine’s durability mode is a choice about which of these to survive. Weakening the commit flush — acknowledging before the flush completes — trades the power-failure case for latency, and the trade is legitimate for bulk loading and for data that can be regenerated. It is not legitimate as a default, because the failure it permits is silent: the transaction was acknowledged, the application moved on, and the record is simply absent after restart.