Checksum Verification and Restore Drills

A backup job that exits zero has demonstrated one thing: the software did not encounter an error it knows how to report. It has not demonstrated that the data reached the destination intact, that the destination can still read it, or that the data is the data that was wanted. Those are three separate claims, they are tested by three different operations, and the word verification is used for all of them.

The distinction matters because the cheapest of the three is the one most often implemented, and it is the one that proves the least.

Three questions that share one word

The first question is whether the repository’s structure is sound. Its metadata parses, every snapshot references packs that exist, every index entry points at an offset inside its pack, and the reference counts are consistent. A check that answers only this question reads a small fraction of the stored bytes, finishes quickly, and catches the failures caused by interrupted writes, crashed processes, and half-completed deletions. It cannot catch a flipped bit inside a pack, because it never reads the pack.

The second question is whether the stored bytes are the bytes that were written. Answering it requires reading every object and recomputing its digest, comparing against the identifier recorded when the object was stored. This is what content-addressed storage makes possible: the identifier is the hash, so verification is a matter of rehashing and comparing, with no separate manifest to trust. The cost is a full read of the repository, which for a large store on object storage is also a full egress and request charge.

The third question is whether a restored system works. No amount of hashing answers it, because the data can be bit-perfect and still be the wrong data, unusable without a key, or incomplete in a way the backup software never modelled. Only a restore answers it, and only a restore into something that resembles the target environment answers it usefully.

Most backup tools expose the first two as separate commands, and the distinction is worth knowing before scheduling either. A frequent arrangement runs the cheap structural check after every backup and the expensive data check on a slower cadence, because the expensive one competes with the backup for the same I/O and, in some designs, the same lock.

What a hash covers

A cryptographic hash maps an arbitrarily long input to a fixed-length output: 256 bits for SHA-256, 256 bits for BLAKE2s, 512 for BLAKE2b. The property that matters for corruption is that a change to any input bit changes the output with overwhelming probability. The property that matters for an adversary is collision resistance, which for a 256-bit hash is about 128 bits by the birthday bound.

Those two properties are not the same requirement, and conflating them leads to over- or under-engineering. Detecting accidental corruption, the kind produced by a failing disk or a truncated transfer, needs only a good non-cryptographic checksum: CRC-32C detects all burst errors up to 32 bits and is cheap enough to compute in hardware. Detecting deliberate substitution needs collision resistance, because an attacker who can write to the repository can replace an object and recompute a CRC.

Neither property provides authenticity. An unkeyed hash computed by the storage system proves that the bytes read are the bytes that were hashed by that same system. If the attacker can write both the object and its recorded digest, the check passes. Authenticity requires a key: an HMAC over the object, or a signature over the repository’s root manifest, with the key held somewhere the attacker does not control. That is a different design from integrity checking, and most backup tools leave it to the storage layer or to an explicit signing feature.

There is also a layer question. A filesystem with per-block checksums, such as ZFS or btrfs, detects corruption when a block is read and can repair it from a replica. A backup stored on such a filesystem inherits that protection while it sits there, but the protection does not travel with the data: reading the backup through the filesystem returns corrected bytes, and copying those bytes elsewhere leaves them unprotected. The end-to-end hash computed by the backup tool is the only check that survives the move.

How much sampling is enough

Reading the entire repository is expensive, so the natural compromise is to read a random subset. How large the subset must be is a question with an answer, and the answer is usually larger than expected.

If a fraction p of objects are damaged and n objects are sampled independently, the probability that none of the damaged objects is sampled is (1 - p)^n. To be 95 percent confident of finding at least one damaged object, n must satisfy (1 - p)^n ≤ 0.05, so n ≥ ln(0.05) / ln(1 - p).

For p = 0.01, one percent of objects damaged, that gives about 299 samples. For p = 0.0001, one object in ten thousand, it gives about 30,000 samples. The second number is not a sampling strategy; it is a full scan with extra steps, because a repository with 30,000 objects is not large and the sample covers all of it anyway.

The practical conclusion is that sampling detects broad damage, such as a failed migration that truncated many packs, and cannot detect sparse damage, such as a single corrupted object. Sparse damage is exactly what a failing disk produces. Sampling is therefore a compromise for the period between full scans, not a substitute for them.

The sample must also be drawn from the right population. Sampling objects uniformly is not the same as sampling snapshots uniformly: a snapshot with a large share of the repository’s unique data has a proportionally larger chance of being hit, and a snapshot that shares almost everything with its predecessor has almost none. A scheme that samples a fixed number of objects per snapshot over-weights small snapshots and can miss the one large one entirely.

The failures a checksum cannot see

The set of failures that survive a bit-perfect verification is larger than it first appears.

The backup can be of the wrong thing. A mount point that was not mounted when the job ran produces a perfectly consistent, correctly hashed, empty backup. A database whose data directory moved produces a backup of the old, now-unused directory. A path that was excluded by a pattern the administrator thought matched something else produces a backup missing exactly the files that mattered. In all three cases the hashes match, because the backup software recorded what it saw.

The backup can be consistently corrupt. If an application writes a malformed record and the backup faithfully copies it, every checksum agrees. Corruption that originates above the storage layer is invisible to storage-layer verification, which is why restore drills check application-level invariants such as row counts, referential integrity, and the ability to run a query.

The backup can be unreadable for reasons outside the data. An encrypted repository whose key was rotated away, a tape whose drive model is no longer manufactured, a snapshot whose parent was deleted by a retention policy that did not understand the chain: none of these change a single stored byte, and all of them make the data unrecoverable. The check that matters for these is not a hash but a restore, performed before the failure.

Finally, the check can be lying about its coverage. A verification job that runs against a replica which the backup job does not write to verifies the wrong copy. A verification job that shares credentials with the backup job verifies whatever the compromised credential can see. A verification job that reads through a cache can verify the cache.

Drills that produce a number

A restore drill exists to produce a duration and a pass or fail, not a feeling of reassurance. The useful form is a restore of a named backup, chosen deliberately, into an environment that is isolated from production and configured like the target. The drill records the wall-clock time from “start” to “service answers a real request”, because that is the number the recovery time objective is measured against.

Two choices in the drill change what it proves. Restoring the most recent backup tests the common path and the shallowest chain; restoring the oldest backup retained by policy tests the deepest chain and the retention policy’s assumptions at the same time. Restoring into the real environment tests the runbook; restoring into a clean environment tests the runbook plus every dependency the runbook assumed was already present.

The drill also has to test the parts that are not data. Credentials to reach the backup store, decryption keys, DNS names, TLS certificates, and the identity provider all have to exist for the restored system to be reachable, and all of them can expire independently of the data. A restore that succeeds technically and produces an unreachable service has found a real defect.

Recording the result turns the drill into evidence. A restore time that is not measured is not known, and a chain whose depth has grown since the last drill has a restore time that has grown with it. The trajectory is the useful signal: the same drill run quarterly, against the same class of backup, shows whether the recovery time objective is still met or is being met by luck.

The dependency on keys is what makes key escrow for encrypted backups part of the drill rather than a separate concern, and the ordering constraints that a full estate imposes on a restore are covered in recovery ordering for multi-service estates.