Scrub Passes: Finding Latent Damage Before It Is Needed

A checksum only helps if something reads the block and compares. In normal operation, most blocks are never read: an archive that was written once and is read twice a year has almost no verification coverage, and the first read after a year may be the one that discovers a fault. Scrubbing exists to move that discovery from the moment of need to a moment when there is time to act on it, and the cost of that move is a full read of the medium.

Latent damage is the normal case, not the exception

Media faults that survive a write are silent until the block is read. A platter sector whose magnetic domain weakens over months, a NAND page whose charge leaks toward a neighbouring level, a bit that flips from a cosmic ray while the drive is idle — all of these are present in the stored data and invisible to every structure above it. The device’s own ECC may be able to correct them, in which case the read succeeds with a correctable-error count; if the degradation has passed the ECC’s correction capability, the read fails; if the degradation is in a region the ECC does not cover, or the error is in the path rather than the medium, the read may succeed with wrong bytes.

The important number is the width of the window between damage occurring and damage being discovered. Without scrubbing, the window is bounded below by the time since the write and above by the time until the next read of that block, which for cold data can be years. With scrubbing, the window is bounded by the interval between passes. That interval is a policy choice, and the useful way to make it is to compare it against the time it takes redundancy to become inadequate.

Consider a two-way mirror. One copy can be lost without data loss. If a block is damaged on one device and undetected, and the other device fails entirely before the damage is found, the data is gone. Scrubbing shortens the period during which a single undetected fault is one failure away from data loss. This is why scrubbing matters far more on redundant layouts than on single-device ones: on a single device there is no repair path, so scrubbing converts an undetected fault into a detected one but cannot fix it.

Why the pass walks metadata

A linear sweep of the device — read every sector from zero to the end, verify what can be verified — is the obvious design and the wrong one. A checksum stored in a block pointer is not adjacent to its block, so a raw sweep cannot verify data at all; it can only verify structures that carry their own checksums, such as metadata blocks in ext4 or btrfs tree blocks. Worse, a raw sweep reads free space, which has no checksum and no owner, and it reads blocks belonging to snapshots and to files whose pointers have since been rewritten.

A filesystem scrub instead traverses the tree from the root, visiting every reachable block in logical order, verifying each against the checksum recorded in the pointer that named it, and reading the data blocks at the leaves. The traversal is the mechanism, and its properties follow from the tree:

Every allocated, reachable block is visited exactly once. Blocks that are allocated but unreachable — leaked space — are not visited, which is a defect in the allocator rather than in the scrub. Free space is not read at all.

Verification happens with the correct expectation, because the pointer supplies the checksum, the compression type, the encryption parameters, and the birth time. A raw sweep would have none of these and would need to reconstruct them.

The traversal can be paused and resumed at a node boundary, which makes a scrub interruptible without leaving a partial state.

The traversal is also the cost. Reading the tree means reading metadata blocks that are already in cache and would not otherwise be read, and it means the read pattern is scattered rather than sequential. On a spinning disk, a scrub of a large filesystem is dominated by seek time rather than transfer time; on flash, by the random-read throughput of the device. Both are far below the sequential bandwidth the device can sustain, which is why scrub duration is usually measured in hours per terabyte rather than in minutes.

/* Depth-first scrub over a block tree. Each child is verified against
   the checksum carried by the pointer that named it, before descent. */
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

uint32_t crc32c(const void *buf, size_t len);   /* the filesystem's checksum */

struct bp { uint64_t addr; uint32_t len; uint32_t crc; uint8_t is_leaf; };

/* Policy hook: log the damaged block and return a per-pass error summary. */
int damage_recorded(const struct bp *bp, int depth);

int scrub_node(int fd, const struct bp *bp, int depth)
{
    uint8_t *buf = malloc(bp->len);
    if (!buf)
        return -1;
    if (pread(fd, buf, bp->len, bp->addr) != (ssize_t)bp->len ||
        crc32c(buf, bp->len) != bp->crc) {
        free(buf);
        return damage_recorded(bp, depth);   /* report the block, keep going */
    }
    if (bp->is_leaf) {
        free(buf);
        return 0;
    }

    int rc = 0;
    for (size_t i = 0; i < bp->len / 17; i++) {   /* 17-byte pointer records */
        const uint8_t *p = buf + i * 17;
        struct bp child;
        memcpy(&child.addr, p, 8);
        memcpy(&child.len, p + 8, 4);
        memcpy(&child.crc, p + 12, 4);
        child.is_leaf = p[16];
        rc |= scrub_node(fd, &child, depth + 1);
    }
    free(buf);
    return rc;
}

Two details in that sketch carry most of the behaviour. Damage is recorded and traversal continues, because aborting the pass on the first error leaves the remaining blocks unverified and the operator with a single data point. And the recursion descends only after the parent verifies, so a corrupted pointer is never followed.

Throttling, and what a scrub competes with

A scrub is background work that reads the entire medium. Run at full speed it saturates the device and starves foreground reads, turning a maintenance operation into an outage. Every implementation therefore has a throttle, and the throttle’s design determines whether the scrub finishes in a week or in a month.

The crude form is a delay between I/O operations, which produces a fixed duty cycle independent of the load. The better form measures the device’s queue depth or recent latency and issues scrub I/O only when the device is otherwise idle. Implementations differ in which signal they use: some track the block layer’s queue, some measure the latency of their own scrub reads and back off when it rises, some expose a pause and resume interface for the operator to drive.

There is a real tension here that no throttle resolves. A scrub that yields to foreground I/O never completes on a busy system, because the system is always busy. A scrub that does not yield causes the latency spike it was throttled to avoid. Practical designs accept a floor: a minimum rate that the scrub maintains regardless of load, sized so that a pass completes within the intended interval even on a loaded system. The operator-facing contract is usually expressed as a duration or a rate rather than as a priority, because the duration is what determines the detection window.

What a scrub can and cannot report

The output of a scrub is a set of verified blocks, a set of blocks that failed verification, and — on redundant layouts — a set that was repaired. Those three categories have very different implications.

A repaired block is a success story with a cost: the repair consumed a write, and if the redundancy was degraded (a mirror with one device already failed), the repair may not be possible. A filesystem that reports a repair has also learned something about the device, and the count of repairs per device is the signal that predicts failure.

A failed block with no redundancy is data loss, reported as such. The filesystem can say which file and which offset, which is far more useful than a device-level bad-block report, because the operator can restore that file from backup rather than replacing the device.

A block that verifies is not proof of correctness. The checksum proves the block matches what was written; it says nothing about whether what was written was right. A scrub therefore cannot detect a corruption that was present at write time — a bug in the application, a bad page read into the buffer cache, a misdirected write whose checksum was computed for the destination address. Scrubbing finds damage that occurred after the write, which is the majority of media faults and none of the software ones.

That limit is why scrub frequency is not a substitute for backup, and why the checksum’s placement and seeding matter so much. How large a region each checksum covers decides which faults it can localise and what a repair costs, which is the trade worked through in checksum granularity and the read-modify-write penalty. A checksum that includes the block’s address in its input catches a misdirected write when the block is later read; a checksum that omits the address offers no such protection, and checksum blind spots and misdirected writes works through the cases that survive a clean scrub. The complementary mechanism, reading from a second copy when the first fails, is self-healing reads and reconstructing a damaged copy; scrubbing is what makes that mechanism useful by finding the faults before they are needed.

Scheduling against the failure model

The interval between passes should be shorter than the time over which a second failure becomes likely, and that time is not a constant. It depends on the device population’s age, the redundancy level, and the rebuild time — because the window of maximum vulnerability is the period during which a failed device is being rebuilt, when the array is running with less redundancy than designed.

A practical policy therefore ties the scrub interval to the rebuild time: scrub often enough that a full pass completes well inside the rebuild window, so that a rebuild never runs against data that has not been verified since the last device loss. On a large array whose rebuild takes days, that argues for a pass measured in weeks rather than months. On a small mirror, a monthly pass is ample. Neither number is a universal constant, and a filesystem that lets the operator set the interval is acknowledging that the answer depends on hardware the filesystem cannot see.