Torn Writes, Sector Atomicity, and the RAID Write Hole

A filesystem writes a 4096-byte metadata block and the power fails during the transfer. After recovery, the first 2048 bytes hold the new version and the last 2048 hold the old one. The block’s own length field says it is a valid block, the pointer that names it is valid, and the tree above it is intact. Nothing about the structure is broken except that half of it belongs to a state that never existed. A checksum over the block catches this. A filesystem without one does not, and its recovery will happily walk the mixture.

The atomic unit is the logical sector, and it is smaller than the block

Drives have historically exposed 512-byte logical sectors and 512-byte physical sectors, and a write of one logical sector has been atomic in the sense that the drive either programs it fully or leaves the previous contents. That guarantee is a property of the interface, not of the media. A modern drive with 4096-byte physical sectors and 512-byte logical emulation implements an atomic 512-byte write by reading the containing 4096-byte physical sector, modifying part of it, and programming the whole thing. If power fails during that programming, the emulation cannot guarantee the 512-byte write landed, because the physical operation was 4096 bytes.

The industry response has been the 4Kn format: a drive that reports 4096-byte logical and physical sectors, where a write of 4096 bytes is the atomic unit and a write of 512 bytes is not allowed. That removes the emulation hazard for 4 KiB writes, at the cost of requiring the filesystem’s block size to be at least 4096 bytes and of making 512-byte metadata structures awkward to store. Most filesystems moved to 4 KiB blocks for this reason, and the ones that did not rely on read-modify-write in the drive, which is exactly the emulation that 4Kn removes.

The practical rule is that atomicity is guaranteed only for writes that are aligned to and no larger than the device’s reported logical sector size, and that a device which reports 512 bytes while physically programming 4096 bytes has a gap between what it promises and what it can deliver under power loss.

Torn writes break assumptions silently

A torn write is a write that lands partially. It is most likely when the write spans the boundary between two media units, two flash pages or two disk sectors or two stripe members, because the device may complete the first unit and lose power before the second. It is also likely when the device is doing an internal read-modify-write, as described above.

The failure mode is not that the block is unreadable. It is that the block is readable and wrong. A tree node whose first half is new and second half is old still has plausible child pointers; a directory block whose first entries are new and last entries are old still has a valid record structure; a log record whose header is new and payload is old still claims a length and a checksum that may or may not match. Recovery code that trusts structure will follow these mixtures into states that no correct execution produced.

Two defences exist and they are usually combined. The first is a checksum over the block, stored outside the block or in a header that is itself covered, so that a torn block fails validation and is treated as absent. The second is to write the block to a new location rather than in place, so that the old copy remains valid and the pointer can be left pointing at it, which is the shadow-paging technique described in Shadow Paging and Copy-on-Write Trees.

Checksums have their own limits. A 32-bit CRC over a 4 KiB block detects any single-bit error and any burst shorter than the polynomial’s degree with certainty, and detects arbitrary errors with probability about 1 in 2^32. That is adequate for random media errors and for most torn writes, because a torn write produces a structured difference rather than an adversarial one. It is not adequate against a device that returns a block from the wrong location entirely, which is why systems that care also store the block’s identity, its logical address or its object number, inside the checksum’s coverage.

Parity makes every write a read-modify-write

A RAID 5 array stores data across N-1 members and one parity block per stripe, computed as the XOR of the data blocks. Updating one data block requires updating the parity, and the parity is the XOR of all the blocks, so the array must either read all the other data blocks and recompute, or read the old data and the old parity and apply the difference. The second is cheaper: new parity equals old parity XOR old data XOR new data, which needs two reads and two writes instead of N-1 reads.

The window between the data write and the parity write is the write hole. If power fails after the data block is written and before the parity block is written, the stripe is inconsistent: the parity no longer matches the data. A later read of the modified data block succeeds, because it reads the data directly. A later read of a different block in the stripe may need the parity to reconstruct it if that block’s member has failed, and the reconstruction will produce a wrong value. The array’s redundancy has been silently consumed.

Worse, the array cannot tell which block in the stripe is wrong. Parity says only that the XOR of all blocks is nonzero; it does not localize the error. A scrub can detect the inconsistency but cannot decide whether the data block or the parity block is stale, so it cannot repair it without additional information. That is the hole: an inconsistency that redundancy cannot resolve because the redundancy is the thing that is inconsistent.

Closing the hole

The standard fix is a write-intent log, usually a small journal on a separate device or a dedicated region. Before writing a stripe, the array records the stripe number and the new data in the journal; after the data and parity are both written, it clears the entry. On recovery, any journal entry that was not cleared is replayed: the stripe is rewritten from the journal’s copy, which restores consistency regardless of which half landed. The journal converts a multi-block update into a single-block atomic one, which is the same reduction that write-ahead logging performs for filesystems.

The cost is one extra write per stripe update plus the journal’s own flush, and the journal becomes a single point of failure if it is not itself redundant. An array without a journal can still avoid the hole for full-stripe writes, where all data blocks and the parity are written together: if the array can write an entire stripe atomically, there is no window. That is why some arrays buffer partial writes until they accumulate into a full stripe, trading latency for the elimination of the hole.

Copy-on-write arrays avoid the problem differently. Rather than updating a stripe in place, they write new blocks elsewhere and update a pointer, so the old stripe remains intact until the new one is complete. The cost is the same as shadow paging’s: space amplification and a garbage collection pass to reclaim the old blocks. It also makes the array’s write pattern random rather than sequential, which is a poor fit for the media it is usually built on.

Checksums above and below the array

A filesystem checksum and an array checksum protect against different things. The array’s per-block checksum, if it has one, detects a block that the array read incorrectly from a member. The filesystem’s checksum detects a block that the array returned correctly as far as the array knows but which is not the block the filesystem asked for, or which is a torn version of it. Neither subsumes the other, and a system with only one of them has a gap.

The layering matters for recovery. If the filesystem’s checksum fails on a block, and the array has redundancy, the filesystem can ask the array to reconstruct the block from parity. That only works if the array knows the block is bad, which it learns from the filesystem’s report. This is why scrubbing is done at the filesystem level with the array’s repair facility available: the filesystem finds the bad block, the array regenerates it. An array that scrubs on its own can only find inconsistencies it can detect from parity, and as noted, it cannot localize them.

The order of verification also matters. A filesystem that verifies a checksum before following a pointer will refuse to descend into a corrupt subtree, which limits the damage to the subtree’s contents. A filesystem that follows first and verifies later can walk into arbitrary memory or arbitrary disk offsets. This is the reason checksums are stored in the parent’s pointer structure rather than only in the block: the parent can validate the child before using it, which turns a corrupt block into a missing subtree rather than a wild pointer.

What remains after all of this

Torn writes are handled, parity holes are closable, and checksums catch corruption, but the combination still leaves a class of failure that none of them addresses: a device that acknowledges a write it did not perform. A drive with a firmware defect, a virtual disk whose cache is not flushed by the hypervisor, and a controller that reorders a flush behind a write all produce a state where every software guarantee held and the media is still wrong. Detecting that requires end-to-end verification after the fact, reading back what was written, which is expensive and usually done only at scrub time.

The remaining exposure is why the durability chain is only as strong as its weakest acknowledgement, and why a system that needs a guarantee stronger than the device’s must add redundancy above it. The mechanisms for doing that at the filesystem level, and the recovery procedures that make the redundancy usable, are what the last articles in this cluster take up.