What write() Actually Does Before Data Leaves the Process

A successful write of a hundred megabytes to a regular file can complete in a fraction of the time a write of four kilobytes takes to reach the device. Nothing is wrong with either call: the large one did not move data to storage, and the small one may not have either. The call’s contract is to transfer bytes from the process into the kernel, not from the kernel to the medium. Everything that determines whether the write is fast, slow, or durable follows from that split.

The buffer stops being yours at return

For a regular file the generic write path copies the caller’s bytes into page-cache pages belonging to the file’s inode. If the page is already present it is copied into; if not, a page is allocated, and the range that is not fully overwritten must first be read from the device so that the untouched tail holds the file’s real contents. That read-before-write is invisible to the caller and is the reason a small write into a large sparse hole is more expensive than the write size suggests.

The return value is the number of bytes copied into the cache. POSIX requires the call to report how much of the buffer it consumed, and permits a short count for reasons the caller cannot distinguish from end of file; a short write is permitted when a signal interrupts the copy, when the descriptor was opened non-blocking and the destination could not accept more, when the file size limit was reached, or when the medium itself fills before the whole request is written. It never means “the disk wrote this much”.

Whether the copy happens at all depends on the file’s cached state. A page marked for writeback keeps being writable; the new bytes go into the same page and the page stays dirty, so the writeback already scheduled may or may not include the latest data. Nothing in the interface tells the caller which case occurred.

#include <errno.h>
#include <unistd.h>

/* Append a buffer, tolerating short writes; durability is a separate step. */
int write_all(int fd, const void *buf, size_t len)
{
    const char *p = buf;
    size_t done = 0;

    while (done < len) {
        ssize_t n = write(fd, p + done, len - done);
        if (n < 0) {
            if (errno == EINTR)
                continue;
            return -1;
        }
        done += (size_t)n;      /* n may be less than requested */
    }
    return 0;
}

Dirty accounting is the throttle

Each written page becomes a dirty page, counted in two places: per-file and per-backing-device on the kernel side, and in the global dirty counters that the writeback machinery watches. The kernel does not let dirty state grow without bound, because a machine whose entire memory is dirty has nothing left to reclaim from and a crash loses more data than it should.

The policy is expressed as thresholds over total available memory. A background ratio defines the point at which the flusher threads start writing dirty pages out asynchronously. A higher dirty ratio defines the point at which a process that dirties pages is made to wait, because the system is at its limit. Defaults place the background start at ten percent and the hard stop at twenty percent of memory; the exact values are tunable and are frequently raised on machines with fast storage.

Below the background threshold, write cost is a memory copy. Between the thresholds, the copying continues but the flusher is competing for the device, so the latency of a write that triggers allocation rises. Above the dirty ratio, the writing process itself is throttled: it sleeps inside balance_dirty_pages for an amount of time computed from the per-task dirty rate. This is the origin of a well-known shape in write benchmarks, where throughput climbs, plateaus, and then falls sharply once the working set exceeds physical memory and every page must be evicted before its replacement can be used.

Where the pages go next

The flusher writes dirty pages in inode order, collecting physically adjacent pages into larger requests where the filesystem can map them contiguously. Pages that are re-dirtied before their writeback completes are written again in the next pass; pages that are clean can be reclaimed or dropped. Expensive corner cases live here: an inode that is continuously dirtied can prevent its pages from ever completing writeback, and a page shared by two mappings must be flushed from one of them before it can count as clean.

The writeback itself passes through the filesystem and the block layer, where requests may be merged and reordered, and then to the device, where a volatile cache may hold them. Writeback Scheduling and Merging at the Block Layer describes that second half of the path. A write that has reached the page cache has satisfied the interface but has not survived a power loss.

fsync, fdatasync, and the ordering they buy

fsync asks the kernel to flush the file’s dirty pages and, if the file’s metadata changed, the metadata as well, and to wait for the device to report completion. fdatasync omits metadata that is not needed to retrieve the data, which in practice skips an inode update for a write that does not change file size. sync_file_range allows flushing a range without waiting, which is how a program can start device writes early and pay for durability later.

The cost of each is set by the device and by the number of other writes in flight. On a device with a volatile write cache, a flush means a cache flush command, and each one is a round trip that can be a substantial part of a millisecond on a spinning disk. A program that calls fsync after every small write issues one device round trip per record; a program that batches many records and calls fsync once amortises that round trip. The correctness requirement, not the throughput budget, should decide which one is used. Databases generally need the batch to correspond to whatever unit must be atomic after a crash, which is why group commit exists at all.

There is a second durability question that fsync does not answer: whether the directory entry pointing at a newly created file is durable. Creating a file dirties its parent directory, and durability of the file’s contents without the directory entry leaves no way to find them, so the directory must be synced too. This is a distinct object and a distinct call on the directory descriptor.

Ordering, not just completion

POSIX gives write to the same file no ordering guarantee between concurrent writers, but it does guarantee that a write is atomic with respect to a read of the same bytes for regular files as far as the interface is concerned, and it guarantees that writes to a pipe or terminal below a certain size are not interleaved. Those guarantees concern the cached state. Durability ordering across crashes is a stronger property that requires a device that either honours flushes or is written with an interface that bypasses its cache.

A related trap is that a successful fsync on one file says nothing about another file’s data, even if one was written after the other. The classic append-then-rename idiom relies on syncing the temporary file before the rename and syncing the directory after it; skipping either step produces a state in which the new name refers to a file whose contents are absent or stale after a crash. rename itself is atomic with respect to visibility, so a concurrent reader sees either the old file or the new one, but atomicity of the name change is not the same as durability of the data behind it.

Signals, mmap, and the cases where the model breaks

A write to a pipe with no reader raises SIGPIPE unless the process ignores or blocks it, and a socket write blocks or returns EAGAIN depending on flags. A write into a memory-mapped region never calls write at all: the CPU store dirties the page directly, and the dirty accounting happens on the page fault path rather than in a syscall. The same dirty thresholds apply, but the point at which a process can be throttled moves from the syscall to the fault handler, so an application that maps a file and writes to it sees the stall at a different place in its code.

Terminal devices and some character devices do not cache the way files do. A write to a serial port can block until the transmitter has room, and its return means the bytes were handed to the driver’s buffer rather than the wire. Treating the write path as uniform across descriptor types is the source of most incorrect assumptions about when a byte has been “sent”.