Read-Path Verification and What Happens When a Block Fails

Verifying a block on read sounds like a two-line change: compute the checksum, compare, return an error on mismatch. What makes it a design problem is that a verification failure carries almost no information about its own cause. The block may be damaged, the checksum may be damaged, the pointer may point at the wrong block, the read may have come from a stale cache, or the caller may be reading a region that was never written. Each of those needs a different response, and the read path must decide which one it is looking at without any further evidence than a mismatch.

The order of operations a verifying read must respect

Four steps have to happen in a fixed order, and getting the order wrong produces either false positives or silent acceptance.

The block must be fetched from the medium or from a cache that is known to hold the block’s exact contents. A page cache entry keyed by inode and offset is not sufficient evidence on its own, because the cache may have been populated from a block that has since been rewritten at a different physical address. Filesystems that verify generally either bypass the cache for verification or track the physical identity of the cached block.

The checksum must be read from a source that is itself verified. In ZFS the checksum comes from the block pointer inside the parent block, and the parent block’s own checksum was verified when it was read, so the chain of trust is established from the root down. Reading a checksum from an unverified location means a corrupted checksum can cause a good block to be rejected or a bad block to be accepted, and the second is worse.

The recomputation must run over exactly the bytes that the writer covered. This is where most implementation bugs live: a checksum computed over the payload after compression but verified before decompression, or computed over logical bytes but verified over physical sectors, will mismatch on every read rather than on damaged ones. ZFS records the checksum over the logical (uncompressed, unencrypted) content and verifies after decompression, which means decompression runs before verification and a corrupted compressed block can therefore fail in the decompressor rather than in the checksum comparison.

The result must be delivered to the caller only after the comparison, with no window in which the unverified buffer is reachable. A common structure is to read into a private buffer, verify, and only then copy or hand off. Handing the page to the caller and verifying afterwards races with the caller’s use of the data, and on a mismatch there is no way to retract it.

/* Verify-then-publish: the buffer stays private until the check passes. */
#include <stdint.h>
#include <stdlib.h>
#include <unistd.h>

uint32_t crc32c(const void *buf, size_t len);

struct extent {
    uint64_t addr;        /* physical block address */
    uint32_t len;         /* bytes covered by the checksum */
    uint32_t expected;    /* CRC32C recorded at write time */
};

int read_extent(int fd, const struct extent *e, uint8_t **out)
{
    uint8_t *buf = malloc(e->len);
    if (!buf)
        return -1;

    if (pread(fd, buf, e->len, e->addr) != (ssize_t)e->len)
        goto fail_io;
    if (crc32c(buf, e->len) != e->expected)
        goto fail_check;

    *out = buf;
    return 0;

fail_check:
    free(buf);
    return -2;              /* detected damage: caller decides policy */
fail_io:
    free(buf);
    return -3;              /* device reported failure: retry is meaningful */
}

The split between -2 and -3 is the whole point of the exercise. A device-level failure may be transient — a marginal sector that a retry with different timing, a lower transfer rate, or a reallocation recovers. A checksum mismatch is not transient in the same way: the bytes are already in memory and they do not match the recorded expectation. Retrying the identical read returns the identical bytes from the identical location. The only retry that can help is one that reads a different copy, which requires redundancy.

Sorting mismatches into classes

A read path that returns a generic I/O error on mismatch discards information the filesystem already has. Four classes can be distinguished cheaply.

A data mismatch with a valid pointer means the block at the expected address does not match the expected checksum. This is the ordinary damage case: media degradation, a lost write, or a write that landed elsewhere. The response is to try another copy if one exists, otherwise return an error that identifies the affected logical range.

A pointer mismatch means the parent block failed verification, so the pointer itself is suspect. This is the more serious condition, because the pointer may name an arbitrary address. A filesystem that trusts a corrupted pointer can read a block belonging to another file and return it as this file’s data; the read succeeds, the payload verifies against the checksum the corrupted pointer carries, and nothing in the payload’s own check notices. The defence is structural: the pointer lives inside a block that is itself checksummed, so a corrupted pointer is caught one level up, before any address in it is followed.

A generation or birth-time mismatch guards the case where a pointer is intact but stale. ZFS records the transaction group in which a block was born in the block pointer; a pointer replayed from an older tree carries an older birth time than the block now at that address, and the read fails on the comparison rather than on the checksum. Filesystems without an explicit generation field rely on the checksum alone for this case. That works, since an overwritten block has different contents and therefore a different digest, but it costs a full recomputation of the block to establish what a 64-bit field comparison would have shown.

A checksum field mismatch with valid data is the inverse of the first case: the data is intact and the recorded value is wrong. It arises from a corrupted metadata block, a bug in the writer, or a block written by an older version of the filesystem with a different checksum algorithm. This class is indistinguishable from the first without redundancy, and it is why changing the checksum algorithm on an existing dataset requires rewriting every block rather than just changing a setting.

Policy after the mismatch

Once the class is known, the response is a policy decision that the filesystem usually exposes as a mount option, because the right answer depends on what the administrator wants to learn.

Returning an error to the caller is the default for most filesystems: EIO from the read, with the affected range logged. It is honest and it is destructive of availability — a single bad block in a large file makes the file unreadable unless the caller can tolerate partial reads.

Returning zeros or best-effort data is what some systems do to preserve availability. It converts a detectable corruption into an undetectable one from the application’s point of view, which is acceptable only when the alternative is worse and the event is recorded. A database that receives zeros in a page it will checksum itself will usually notice; a program reading a text file will not.

Marking the block bad and continuing is the middle path, and it requires the filesystem to have a place to record the decision. On a mirrored or RAID-Z layout, the failed copy is taken out of service and a repair is scheduled; the read succeeds from the other copy and the caller never learns. That repair path is the subject of self-healing reads and reconstructing a damaged copy.

Rejecting the read entirely is the policy for metadata. A failed metadata block means the tree cannot be trusted to describe the file, so continuing to walk it may produce arbitrary results. Filesystems generally treat metadata verification failure as fatal for the operation and, in some cases, as grounds for refusing to mount.

The cost that verification adds

The measurable cost of read-path verification is the checksum computation, which for CRC32C over a 128 KiB block is a small fraction of the I/O time on any medium faster than a spinning disk. The structural cost is larger: verification requires that checksums be available at read time, which constrains caching (cached blocks must be re-verified or their verification status must be tracked), constrains the read-ahead logic (readahead must carry the pointer’s checksum, not just the address), and constrains the interface between the filesystem and the block layer (the filesystem must be able to ask for a specific physical block and get a failure distinguishable from a transport error).

The most common real defect in this area is not a slow checksum but a path that skips verification. Direct I/O, memory-mapped access, and reads that go through a filesystem’s own page cache are the usual places where a block reaches a consumer without the comparison having run. Each skip is a deliberate trade of integrity for latency, and each one is worth naming explicitly rather than leaving implicit — which is what checksum blind spots and misdirected writes takes up, along with the failure classes that no amount of read-path verification can reach.