Silent Data Corruption and What Device Error Reporting Misses

A read returns a buffer, a byte count, and a status. The status is either success or an error code, and nearly all software treats those two as the complete set of outcomes. Storage devices encourage the assumption, because they carry their own protection: each sector has an error-correcting code, each NAND page has an ECC block, each link in the path has a frame checksum. When those checks fail the device raises a medium error and the read fails loudly.

The gap is that device protection is computed by the device, over data the device received, and validated against data the device later reads back. It answers the question “did this sector survive since it was written” and not the question “is this sector the one the caller asked for”. Everything that can make those two questions differ is invisible to the device, and that difference is the subject of every checksum a filesystem keeps.

Three outcomes of a read, not two

Device error handling is usually described as a two-level hierarchy. Correctable errors are repaired in place by ECC and, depending on configuration, counted and reported later. Uncorrectable errors abort the read: SCSI transports them as sense key MEDIUM ERROR, NVMe as an unrecovered read error in its status field, and the block layer turns them into a failed I/O that the filesystem can retry, remap, or return to the caller.

The third outcome is a read that completes with a success status and returns bytes that are not the bytes that were written. Nothing in the device’s reporting distinguishes it from a correct read, because from the device’s position it is a correct read: the sector ECC validates, the transport CRC validates, the transfer completes. The contents are wrong because they were wrong before the ECC was computed over them, or because the ECC was computed over content that was never the intended content.

The distinction matters because the two failure modes need different machinery. A detected error is an event the storage stack already knows how to route: retry the read, consult a mirror, mark the sector bad, notify the operator. An undetected error produces no event at all. It propagates upward as data and is discovered, if ever, by whatever consumes it — a parser that rejects the input, a compiler that fails on a truncated source file, a database that returns the wrong row. By then the original block may have been overwritten and the evidence is gone.

The corruption classes a device cannot see

Four classes account for most silent corruption, and each one defeats device-level checking for a structural reason rather than an implementation bug.

A misdirected write lands at the wrong logical block address. The device writes a full sector, computes valid ECC over it, and reports success. The intended target keeps its old contents and the unintended target is overwritten with data that is internally consistent. Both sectors now pass ECC. The write was durable, atomic, and acknowledged; it simply went to the wrong place. No amount of per-sector protection can detect this, because the corrupted sector is a perfectly valid sector.

A lost write is acknowledged but never reaches the medium. The usual cause is a volatile write cache that loses power before destaging, or a device that reorders a write past a flush it claimed to have honoured. On a later read the device returns the previous contents, ECC-valid, and reports success. The filesystem’s metadata says the new data is there; the medium says otherwise. This is the failure that write barriers and cache-flush commands exist to bound, and it is also why a checksum recorded in metadata is useful: the checksum describes what should be present, not what is.

A torn write leaves a sector or a larger unit partially updated. Drives are specified to write 512-byte or 4096-byte sectors atomically with respect to power loss, but the filesystem’s unit of update is usually larger — a 16 KiB block, a 4 KiB metadata node, a multi-block tree node. Power loss in the middle of such an update leaves some sectors holding new bytes and some holding old bytes. Each individual sector may still pass its own ECC, so the device reports success for all of them, while the composite unit is a mixture that never existed as a whole.

Latent media damage accumulates after a successful write: a bit flips in a platter sector, an adjacent-track write disturb degrades a neighbour, a NAND page’s charge leaks past the point where ECC can recover it. This class is the one devices do detect, eventually, and only when the block is read. Until then the damage is invisible, which is why periodic verification exists as a separate mechanism from read-time checking.

Firmware and controller defects cut across all four. A drive that returns the wrong buffer because of a bug in its own cache management produces exactly the same symptom as misdirected write, and no amount of ECC design prevents it.

Why a lower layer’s check cannot substitute for a higher layer’s

The end-to-end argument states the principle precisely: a function placed at a low layer can be a performance optimisation but cannot be a substitute for the same function implemented at the layer that actually needs the guarantee. A transport checksum protects the bytes in flight between two endpoints. It says nothing about whether the sender read the right bytes from the medium before transmitting them, or whether the receiver wrote them to the right place afterwards. Those are different faults at different layers, and each layer’s check only covers faults below it.

Applied to storage, the reasoning runs as follows. The device’s ECC covers faults introduced between the write head and the read head, because that is the interval the device controls. The block layer’s protection covers faults between the host and the device, because that is the interval the transport controls. Neither covers the interval between “the filesystem decided these bytes belong in block N” and “the device wrote bytes into block N”, which is where misdirected and lost writes live. Only a check computed by the filesystem, over the exact content it intends to store at the exact address it intends, and verified when that content is read back for a consumer, spans that interval.

This is why filesystem checksums are stored in metadata rather than alongside the data. The checksum must be written in the same atomic operation that makes the block reachable, so that a reader cannot observe a block without also observing the value the writer computed for it. Keeping the checksum next to the payload inside the same 4 KiB sector would be simpler, but it would let a torn update replace both payload and checksum consistently, defeating the purpose. Placement is a design question in its own right, covered in checksum placement in block pointers and metadata trees.

What changes once the filesystem keeps its own checksum

With a per-block checksum recorded in a block pointer, a read becomes a two-part operation: fetch the block, recompute the checksum, compare. The comparison converts an undetected error into a detected one at the layer that knows what the data is supposed to be. The device’s role shrinks to transporting bytes; the filesystem’s role expands to deciding whether those bytes are the right ones.

/* Verify a 4 KiB block against the checksum held in its block pointer. */
#include <stdint.h>
#include <unistd.h>

uint32_t crc32c(const void *buf, size_t len);  /* reflected polynomial 0x82F63B78 */

int read_verified(int fd, off_t offset, void *buf, uint32_t expected)
{
    ssize_t n = pread(fd, buf, 4096, offset);
    if (n != 4096)
        return -1;                  /* the device itself reported a problem */
    if (crc32c(buf, 4096) != expected)
        return -2;                  /* device said success; contents disagree */
    return 0;
}

The return value -2 is the interesting one. It is not a storage error and cannot be produced by any device status code, because the information needed to produce it — the value the writer intended — exists only above the device. Once that distinction is available, the filesystem can do things the device cannot: retry against a second copy, mark the block for repair, or refuse to return the data at all rather than hand a caller plausible-looking garbage.

The cost is proportional to how often verification runs. Computing a CRC32C over 4 KiB costs a few thousand cycles with a table-driven implementation and a few hundred with a hardware instruction, so verifying every read is cheap relative to the I/O itself. Cryptographic hashes cost an order of magnitude more, which is why the choice of function is a real decision rather than a formality, as checksum strength and what CRC32 and SHA-256 guarantee sets out.

Detection is also not the same as correction. A checksum can prove that a block is wrong; it cannot say which of two disagreeing copies is right, and it cannot recover a block whose only copy is damaged. The remaining limits — where verification is skipped, which faults no checksum can catch, and why a filesystem can report damage it is powerless to repair — are collected in checksum blind spots and misdirected writes. The practical consequence of this first layer is narrower and more useful: a successful read status is evidence about the device, not about the data, and any system that needs the second kind of evidence has to compute it itself.