Self-Healing Reads: Reconstructing a Copy From Redundancy

A checksum converts an undetected error into a detected one. Detection alone changes nothing about the data: the block is still wrong, and the caller still cannot have it. What detection enables is repair, and repair requires a second independent source of the same content. Self-healing is the mechanism that uses that source automatically, at the moment the read fails, without waiting for an administrator to notice a log line.

The reason this works at all is that redundancy in a checksumming filesystem is not redundancy of bytes but redundancy of information. The filesystem knows, from the pointer, exactly what the block should contain, because it holds the checksum of the intended content. That knowledge is what lets it decide which of two disagreeing copies is the good one — a decision that plain mirroring without checksums cannot make.

The repair loop

A self-healing read is a retry loop with a source-selection step. In outline:

The read is issued against the first available copy, at the physical address recorded for that copy. The block is fetched and verified against the checksum in the pointer.

On a mismatch, the read is issued against the next copy. Each copy is verified independently, against the same expected checksum. A copy that matches is returned to the caller.

Once a good copy is found, the bad copy is scheduled for repair: the good content is written back over the damaged physical location, and the bad location is either overwritten or taken out of service.

If no copy matches, the read fails with a checksum error that names the logical block and every physical location tried.

The crucial asymmetry is in step 2. Verification is against the checksum recorded in the pointer, not against a checksum stored with each copy. If each copy carried its own checksum, two copies could each verify against their own values while containing different data, and the filesystem would have no basis for choosing. With a single expected value recorded once, at most one copy can match, and a mismatch identifies the specific copy as bad. That is the strongest argument for storing the checksum in the pointer rather than beside the payload, an arrangement checksum placement across pointers and metadata trees compares against the alternatives.

ZFS implements exactly this structure. Its block pointer holds one 256-bit checksum and up to three DV addresses. All copies of the block are verified against that single recorded value, and the read path tries each DV in turn until one verifies.

What repair costs on a mirror

On a two-way mirror, the repair is a write of the damaged block to the device that failed verification. The cost is the size of the block plus the cost of the write itself, and the write is usually scheduled rather than performed inline, so the read that discovered the fault returns as soon as a good copy is found.

The subtlety is that the repair write is itself subject to the same failure model as any other write. It can be lost, torn, or misdirected. A filesystem that repairs inline and immediately considers the block repaired has no way to know whether the repair took; a filesystem that records the repair intent and verifies it on a later pass is stronger. The usual compromise is to issue the repair write with the normal durability guarantees and to trust the write path’s own verification — the device’s ECC for the medium, and the checksum for the next read.

There is also a cost that does not appear in the I/O accounting: the read that failed took a full block transfer that was discarded. A workload with a high rate of repairs pays twice for every repaired block, once for the failed read and once for the successful one. That cost is invisible in normal operation because repairs are rare, and it becomes significant precisely when the device is failing and repairs are frequent — the moment when the system can least afford extra I/O.

What repair costs on parity

Erasure-coded layouts change the arithmetic. On RAID-Z with single parity, a stripe holds N data blocks and one parity block, and the content of any one block can be reconstructed by XORing the other N. On double parity, two blocks can be reconstructed using the two independent parity functions. The filesystem’s job during a self-healing read is to determine which block in the stripe is bad and reconstruct only that one.

Here the checksum’s granularity determines the cost directly, in the way checksum granularity and the read-modify-write penalty sets out. If each block in the stripe carries its own checksum, the filesystem reads all blocks, verifies each, identifies the one that failed, and reconstructs it from the rest — reading the entire stripe to repair one block. If the checksum covers the whole stripe, the filesystem knows only that the stripe is bad and must attempt reconstruction for each position in turn, which is both more reads and more computation.

The other cost is the write. Reconstructing a block and writing it back is a read-modify-write of the entire stripe: the parity must be recomputed from the reconstructed data and the other data blocks, which means reading them again if they are not still in memory, and writing both the repaired block and the new parity. On a wide stripe, one bad 4 KiB block can cost a full stripe read plus two writes. This is why parity layouts are much less forgiving of small random writes than mirrors are, and why the mirror-versus-parity decision is fundamentally a decision about repair amplification rather than about capacity.

/* Reconstruct one block of a single-parity stripe from the others. */
#include <stddef.h>
#include <stdint.h>

/* data[i] are the N data blocks, parity is the XOR of all of them.
   Rebuild data[missing] without touching the damaged device. */
void rebuild_block(uint8_t *dst, uint8_t *const *data, size_t n,
                   const uint8_t *parity, size_t len, size_t missing)
{
    for (size_t i = 0; i < len; i++)
        dst[i] = parity[i];

    for (size_t b = 0; b < n; b++) {
        if (b == missing)
            continue;                    /* the bad block is not read */
        for (size_t i = 0; i < len; i++)
            dst[i] ^= data[b][i];
    }
}

The loop reads N − 1 blocks to produce one. For a stripe of eight data disks, repairing one 4 KiB block reads 28 KiB and writes 8 KiB. A mirror repairing the same block reads 4 KiB from the good side and writes 4 KiB. The ratio is the price of the capacity advantage.

When repair cannot complete

Three conditions make self-healing impossible, and each is worth distinguishing because the operator response differs.

All copies damaged. Two copies, both failing verification, means the filesystem cannot tell which is right — and in fact neither is. If the two copies disagree and both fail, the block is lost; the checksum proves it but provides no recovery. On a mirror, this is the state that follows two independent faults, which is exactly what the scrub interval is meant to keep unlikely.

Redundancy already degraded. A RAID-Z1 array with one failed disk has no redundancy left. A block that fails verification on the surviving disks cannot be reconstructed, because the parity that would have rebuilt it is on the failed disk. A read in this state must fail rather than attempt reconstruction, and a filesystem that attempts it anyway may return plausible garbage: the reconstruction arithmetic succeeds regardless of whether the inputs were correct.

Reconstruction crosses a second fault. Rebuilding a block requires reading the whole stripe. If another block in that stripe is also damaged, the reconstruction produces a wrong result — and the wrong result is silently written back over the damaged block, converting a detected fault into an undetected one. This is the classic RAID rebuild hazard, and checksums mitigate it only partly: the reconstructed block can be verified against the pointer’s checksum before being written, so a bad reconstruction is caught rather than installed. That verification step is the difference between a filesystem that repairs correctly under multiple faults and one that spreads corruption during a rebuild.

/* Reconstructed data is verified before it is allowed to replace anything. */
#include <stdint.h>

uint32_t crc32c(const void *buf, size_t len);
int write_block(uint64_t addr, const uint8_t *buf, uint32_t len);

struct bp { uint64_t addr; uint32_t len; uint32_t crc; };

int repair_block(const struct bp *bp, const uint8_t *rebuilt)
{
    if (crc32c(rebuilt, bp->len) != bp->crc)
        return -1;                 /* reconstruction produced garbage */
    return write_block(bp->addr, rebuilt, bp->len);
}

Without that check, the repair path is a corruption amplifier: the more aggressively a system repairs, the more wrong data it writes. With it, the repair is only as good as the redundancy it drew from, and a failure to verify is reported rather than acted on.

The relationship to scrubbing

Self-healing and scrubbing are two halves of one mechanism, and neither is much use alone. Scrubbing finds damage in blocks nobody is reading; self-healing repairs damage in blocks somebody is reading. A system with self-healing but no scrub only repairs what happens to be accessed, so cold data accumulates undetected faults until the day it is needed. A system with scrubbing but no self-healing detects damage and reports it, leaving the operator to restore from backup a file that could have been repaired in place.

The pairing also determines what the operator sees. With both mechanisms active, a scrub that finds a repairable block repairs it silently and the pass reports a repair count. The count is the diagnostic signal: a rising rate of repairs on one device predicts its failure, and a rate that stays at zero across passes is the evidence that the medium is holding. The final layer — protecting the path between the application and the filesystem, where the checksum does not reach — is end-to-end protection information and the checksum boundary.