Checksum Blind Spots: Misdirected Writes and Lost Updates

A checksum answers one question: are these bytes the bytes that were recorded as belonging here? Every failure that leaves that question answered correctly is outside its reach. The set is larger than it first appears, and the reason is structural — a checksum is computed at one instant, over one buffer, by one component, and the rest of the system’s faults happen at other instants, over other buffers, in other components.

Sorting those faults is useful because each needs a different mechanism, and mistaking a checksum for a substitute for the right mechanism is how integrity designs end up feeling complete while leaving the common failures uncovered.

A checksum records intent, not arrival

The write path computes the checksum, then writes the block, then records the checksum where a later reader will find it. If the write never reaches the medium, the checksum describes content that does not exist. On a later read, the block returns its previous contents, verification fails, and the filesystem reports damage — which is correct but late, and the data is gone regardless.

This is the lost-write class, and its mechanism is cache behaviour. A drive with a volatile write cache acknowledges a write when the data reaches the cache, not when it reaches the platter or the NAND. Power loss between those two moments discards the cache. The drive is behaving as specified; the specification simply does not promise durability at write-acknowledgement time unless a cache-flush command was issued and completed.

The host-side equivalent is the page cache. A write to a file returns once the data is in memory, and the block layer may hold it for tens of seconds before issuing it. An application that treats a successful write as durable has made an assumption the interface never supported. The interfaces that carry a durability promise are explicit: fsync and fdatasync on a file descriptor, O_SYNC or O_DSYNC at open time, and the flush and FUA bits in the storage command set. Each of them costs a round trip and a device cache flush, which is why they are used sparingly and why the temptation to skip them is strong. sync_file_range is not in that set: it starts writeback for a range and can be used to shorten the window before a later fsync, but it does not wait for the data to reach the medium, so treating it as a completion point turns an ordering hint into a false guarantee.

The checksum’s contribution here is diagnostic rather than preventive: it turns a silent lost update into a detected inconsistency at read time. Preventing it requires that the checksum and the data be ordered such that the checksum cannot become durable before the data it describes. In a copy-on-write tree this falls out of the structure, because the pointer carrying the checksum is written after the block it points to and the tree’s atomicity guarantees the pair. In an in-place filesystem it requires a barrier between the data write and the metadata write that records the checksum, and a flush before the transaction is considered committed.

Misdirected writes and the addresses a checksum cannot see

A misdirected write deposits correct-looking data at the wrong address. The block it overwrites had a valid checksum for its own content; that content is now gone, and the new content has a valid checksum for the wrong block’s intent. A reader of the overwritten block recomputes and compares against the checksum recorded in its pointer, and the comparison fails — so a per-block checksum stored in metadata does catch this case, but only for the block that was overwritten. The write’s intended target silently keeps its old contents, and nothing in the destination’s verification notices, because the destination was never read.

Three defences exist and they cover different sub-cases.

A checksum that mixes in something other than the payload — a physical address, an inode number, a block index — catches the write at verification time even when the payload is identical to what was intended, because the digest depends on more than the bytes. This is why ext4 seeds its metadata checksums with the filesystem UUID plus the inode number and generation for inode-owned structures, and with the group number for group descriptors: a metadata block copied into another inode’s slot fails verification even when its payload is byte-identical. A scheme that binds the digest to the physical block address gains the same protection against relocation, but must recompute the checksum whenever a block moves.

A reference tag in protection information catches the write at the device, before it is committed, because the tag the host supplied does not match the address the command names. That is the mechanism described in end-to-end protection information and the checksum boundary, and it is the only one of the three that can reject the write rather than detect its consequence.

A generation or birth value in the pointer catches the case where a stale pointer from an older tree names a block that has since been reused. The pointer’s recorded birth transaction does not match the block’s, so the read fails before the content is trusted.

None of the three catches a write that is correctly addressed and contains the wrong data. That fault originates above the storage layer and is discussed below.

Corruption above the filesystem

The most common integrity gap is not on the medium at all. It is the buffer between the application and the filesystem, and the several copies the data makes on its way through the kernel.

A device driver that reads a page into a buffer, a DMA engine that writes to a physical address computed from a stale mapping, a page cache entry that is modified in place by a process with write access to a shared mapping, a copy that crosses a page boundary incorrectly — all of these produce wrong bytes in a block that the filesystem then checksums correctly and writes durably. The checksum is right; the data was wrong when it was checksummed. Every layer below has done its job.

Memory-mapped access deserves particular attention because it bypasses the read and write paths entirely. A store to a mapped page modifies the page cache directly, with no system call, so there is no point at which the filesystem can verify or intercept. The page is later written back by the kernel’s writeback machinery, which computes the checksum over whatever the page contains at that moment. A process that corrupts its own mapping produces a durable, checksummed, wrong block. Detection falls entirely to whoever reads it later, and if the application is the only reader, it may never be detected at all.

The defence for this class is a check computed at the boundary of the data’s meaning: a checksum over a database page, a digest in a file format, a MAC over a message. Those checks live above the filesystem because the faults live above the filesystem, which is the end-to-end argument applied to the layer that actually owns the invariant.

Discard and the shrinking of evidence

Discard operations — TRIM on SATA, the Dataset Management command on NVMe — tell the device that a range of blocks is no longer needed. The device may use that information to erase NAND pages ahead of time, to improve wear levelling, or to stop refreshing a region. The operation is advisory, and its effect on the stored data is unspecified: a device may return zeros for a discarded range, may return the old contents, or may return an error.

The interaction with checksums is easy to get wrong in two directions. A filesystem that discards a block and later reads it without expecting the discard will find contents that fail verification, and may report media damage for a block that was deliberately invalidated. A filesystem that discards blocks belonging to a snapshot, believing them free because the live tree no longer references them, destroys the snapshot’s data — the discard is not aware of the reference counting that made the block shared. This second case is a real hazard in any copy-on-write filesystem with snapshots, and it is why discard must be driven by the allocator’s view of what is genuinely free, not by the tree’s current reachability.

The reverse hazard is subtler. A scrub pass that runs after a discard will read blocks the device has been told to forget. On flash, reading a discarded block can return data that the device is in the middle of erasing, and the ECC may fail — producing a burst of reported damage that is an artefact of the scrub rather than a fault. Filesystems that support both discard and scrubbing generally track which ranges are discarded and skip them, at the cost of a per-extent flag that must itself be crash-consistent.

/* The checksum must cover the bytes actually written, not the caller's
   buffer at the moment the request was built. */
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

uint32_t crc32c(const void *buf, size_t len);
int record_checksum(uint64_t addr, size_t len, uint32_t crc);

int stage_and_write(int fd, const void *user, size_t len, uint64_t addr)
{
    uint8_t *stage = malloc(len);
    if (!stage)
        return -1;

    memcpy(stage, user, len);          /* snapshot the caller's bytes */
    uint32_t crc = crc32c(stage, len); /* hash the staged copy */

    if (pwrite(fd, stage, len, addr) != (ssize_t)len) {
        free(stage);
        return -2;
    }
    free(stage);
    return record_checksum(addr, len, crc);
}

Hashing the caller’s buffer directly and then writing from the page cache is the classic version of this bug: the two operations may see different bytes, because the buffer is writable and the write path may copy it later. Staging first costs one copy and removes the window.

What the residual risk actually is

With a per-block checksum in metadata, a verifying read path, a scrub, and redundancy, the residual fault set is: writes that were correctly addressed, correctly checksummed, and contained wrong data; writes that were acknowledged and lost despite a completed flush; and corruption introduced above the filesystem. The first and third are application-layer problems. The second is bounded by the storage stack’s ordering guarantees, which are the one part of the system that no amount of checksumming can strengthen.

That is why the useful way to think about data integrity is as a set of independent checks, each covering the faults in its own interval. The medium’s ECC covers the medium. Protection information covers transport and addressing. The filesystem’s checksums cover content after the write. The application’s checksums cover meaning. Removing any one of them leaves a class of fault undetected, and adding a stronger function to one of them does nothing for the classes the others cover — which is the practical form of the argument that a lower layer’s check cannot substitute for a higher layer’s, and the reason silent data corruption and what device error reporting misses starts where it does.