Durable Append: Group Commit, Torn Tails, and Recovery

A log file that survives a crash is not the same object as a log file that survives a crash with every record intact. The difference is a set of ordering guarantees that the filesystem does not provide by default, and the cost of obtaining them is dominated by one operation: the flush that forces data from the page cache to the storage device. Understanding what that flush orders, and what it does not, determines whether a recovery procedure can trust the last record in the file.

What write() promises and what it does not

write() copies bytes from the process into the kernel’s page cache and returns. The data is visible to other readers immediately, and it is not durable. A power loss before the page cache is written back loses the data, and the kernel may write it back at any point after the call returns, in any order relative to other writes.

For appends, POSIX adds a specific guarantee: if the file was opened with O_APPEND, the offset positioning and the write are atomic with respect to other writers. Two processes appending to the same file cannot interleave their bytes within a single write() call. What they can do is interleave at the granularity of calls, so a record larger than what one write() submits, because of a short write or because the producer loops, can be split by another writer’s data. A single write() of the whole record, on a descriptor opened with O_APPEND, is therefore the only portable way to get an uninterleaved append.

There is a size limit worth knowing. Writes to a pipe are atomic only up to PIPE_BUF, which POSIX requires to be at least 512 bytes and which is 4096 on Linux. A record exceeding that can be interleaved with another writer’s bytes on a pipe even though it was submitted in one call. Regular files with O_APPEND do not have this limit in the same way, but the atomicity guarantee is a filesystem property, and not every filesystem honours it for large writes.

fsync() is the operation that converts a completed write() into durable state. It flushes the file’s data and the metadata needed to retrieve that data. fdatasync() flushes the data and only the metadata required for retrieval, which avoids a metadata update for fields such as modification time; the difference is one metadata write per flush, which matters when flushes are frequent.

What fsync does not guarantee is ordering between two files. Writing a record to a log and then updating an index in a second file, with an fsync on each, gives no guarantee about which lands first unless each flush is issued and completed in sequence. The sequential form, flushing the log and waiting before writing the index, is the only portable ordering primitive, and it is what write-ahead logging depends on.

The cost model of a flush

A flush that reaches a device with a volatile write cache and no battery has to wait for the device to acknowledge that the data is on stable media. That round trip is the dominant term: the data transfer itself is small, and the wait is the device’s internal write and program cycle. The magnitude varies by orders of magnitude between media, so a spinning disk, a SATA SSD, and an NVMe device with a power-loss-protected cache are not comparable. Any statement about flush latency has to name the medium.

The consequence for a logging path is that a flush per record caps throughput at one record per flush latency, which is far below the rate at which records are produced. A stream of thousands of records per second cannot afford a flush per record on any medium, which forces batching.

There is a second-order effect that is easy to miss: a flush also forces the filesystem to write back whatever else it has dirty, because the ordering requirement covers metadata that precedes the flushed data. A flush issued in the middle of a burst of unrelated file activity therefore does more work than the caller expects. Filesystems that support a range-limited flush reduce this, but the guarantee is narrower and its availability is a property of the filesystem, not of POSIX.

Group commit

Group commit is the resolution: many appends share one flush. The writer accumulates records, and instead of flushing per record it flushes when either a size threshold is reached or a time threshold expires. Every record that arrived during the accumulation is covered by the single flush.

The latency consequence is that a record’s durability latency is the time until the next flush, not the time to flush its own bytes. With a time threshold of T, the worst-case latency is T plus the flush duration, and the average is T/2 plus the flush duration, assuming arrivals are uniform. Throughput becomes the size of the group divided by the flush duration, which grows with the group size until the device’s bandwidth saturates.

The trade is explicit: a larger window raises throughput and raises the worst-case latency for the last record in the group. There is no setting that gives both, and the parameter is chosen from the requirement that actually applies. A system that must not lose more than a bounded amount of work on crash picks the window from that bound; a system that must acknowledge a write to a client before replying picks the window from the client’s latency budget.

A refinement is to decouple the two thresholds. A size threshold alone produces a flush only when traffic is heavy, so a quiet stream leaves records unflushed indefinitely. A time threshold alone produces flushes of a single record when traffic is light, which wastes the device’s capability. Using both, flushing on whichever threshold arrives first, gives bounded latency at low rates and bounded flush count at high rates, which is the shape every practical implementation uses.

The second refinement is to make the flush asynchronous with respect to the writer. The writer submits records to a buffer and returns; a background thread performs the flush and signals completion. Records that must be acknowledged individually then wait on a condition variable tied to the flush generation that covers them, which means a record’s wait is the remaining time in the current flush, not a fixed interval.

Torn tails and why checksums are per record

A crash during a write can leave a partially written record at the end of the file. The bytes written are a prefix of the intended record, and the length field, if the format has one, may be present and correct while the payload is short, or the length itself may be truncated.

A reader that trusts the length field will attempt to read past the end of the file, or will read the beginning of what was meant to be a later record. A reader that trusts a newline delimiter will accept the partial record as complete, because the delimiter may have been written even when the payload was not. Neither failure is detectable from the structure alone.

The standard defence is a per-record checksum over the length and the payload, stored with the record. On recovery, the reader validates each record in turn; the first record that fails validation terminates the scan, and everything after it is discarded. The reason to stop rather than skip is that a corrupted record’s length field cannot be trusted, so the position of the next record is unknown. Stopping is the only safe policy, and it costs the records that follow the corruption even though they may be intact.

The checksum choice matters for the failure it can detect. A simple sum or an exclusive-or detects single-bit errors poorly and misses byte transpositions. A CRC detects all single-bit errors and all burst errors up to its width, which is why CRCs are the conventional choice for on-disk framing. A cryptographic hash detects essentially any change but costs more per byte and is unnecessary when the threat is media corruption rather than an adversary. A keyed hash is the right choice only when the log is an audit record that an attacker might alter.

There is a subtlety about what the checksum covers. Covering the payload but not the length leaves the length unprotected, so a corrupted length can cause a reader to validate a checksum over the wrong span, which usually fails but can, for short records, coincide. Covering the length and the payload together removes that class.

The recovery procedure

Recovery is a scan, and its cost is proportional to the file size, which is why large logs are segmented. Each segment has a header identifying its position in the sequence, and recovery validates segments in order, discarding from the first invalid record onward.

The procedure has three cases.

Clean shutdown. The last segment ends at a record boundary and all checksums validate. Recovery opens the segment for append at the end.

Torn tail. The last record fails validation. The segment is truncated to the end of the last valid record. Records after the torn one, if any, are lost; in an append-only log with a single writer there are none, because the torn record was the last write.

Corruption in the middle. A record before the end fails validation. This indicates media damage or a write that landed out of order, and the remaining records are unreachable through a sequential scan. A format that stores a per-record offset in a separate index can still find the later records, at the cost of a second structure that must itself be crash-consistent.

The last case is the one that motivates the segment-plus-index layout. A sequential format is simple and its recovery is linear, but a single bad sector makes the remainder of the file unreadable. A segmented format bounds the loss to one segment, which is the reason logs are written in fixed-size segments rather than as one growing file. The segment size is a trade between the number of files, the cost of recovery, and the granularity of loss.

Durability and retention meet at this point. A retention policy that deletes whole segments is trivially crash-consistent, because deletion is a directory operation and the remaining segments are untouched. A policy that deletes individual records within a segment requires a rewrite, and a rewrite interrupted by a crash leaves a segment in an intermediate state that recovery must recognize. The retention arithmetic in Retention Policy as a Storage Budget assumes the cheap form; the expensive form needs its own atomicity story.

The blocks that the write path eventually produces are compressed and laid out as described in Block Compression and Columnar Layout for Log Storage, and the flush that makes them durable is the same flush that sets the visibility delay for queries described in Inverted Index Construction and Query Planning for Log Search.