Executing Provenance Records to Certify That a Result Still Reconstructs

A stored provenance record answers the question of what was run. A re-executed provenance record answers the question of whether it still can be. The difference between those two is the difference between a claim that has aged and a claim that is being tested, and the test is what separates a reproducibility statement from a reproducibility practice.

The mechanism that performs the test is a replay loop: take the recorded graph, run each step in a fresh environment, hash the outputs, and compare against the digests stored at creation. Run it every time the inputs change and on a schedule, and the result is a computation whose reproducibility is a measured fact rather than an assumption.

The replay loop as a verifier, not a rebuild

A naive re-run rebuilds everything. That is correct and wasteful. A replay loop is a verifier, and its central operation is deciding, for each step, whether the step’s recorded inputs and code still describe the step, and whether running it still produces the recorded output.

The decision decomposes into three checks per node.

The inputs match. Each input file’s current digest equals the digest recorded in the node. A mismatch means the input changed and the step is stale; the output is no longer evidence about the current inputs.

The code and environment match. The command, the parameter values, and the environment identity equal the recorded ones. A changed script with the same command line is a different computation and must not be replayed as the old one.

The output matches. The freshly produced output digest equals the stored output digest. This is the certification step, and it is the only one that actually verifies reproducibility rather than merely re-running.

Only the last check needs a fresh execution. The first two are hash comparisons and environment lookups, cheap relative to the compute. A replay system that re-executes every step every time is a rebuild tool; a replay system that skips steps whose inputs, code, environment, and stored outputs all still agree is a verifier, and the skip is what makes continuous replay affordable for computations whose individual steps are expensive.

The skip introduces a subtlety: a step is skipped because its recorded output is assumed to be correct for its recorded inputs. That assumption is exactly what bit rot attacks, so a verifier that never re-executes is really a cache. The schedule has to include periodic forced re-execution — even when nothing changed — to detect corruption that hash checks alone would find only if the stored output is compared to a freshly produced one.

What can be certified, and what cannot

A replay loop can certify the property that matters: given these inputs, this code, this environment, the step produces this output. It certifies computational reproducibility, in the narrow sense of Reproducibility, Replicability, and the Boundaries Between Them.

It cannot certify that the input data are correct, that the code implements the method described in the paper, or that the result means what the abstract claims. Those are judgments the loop cannot make, because they require comparing the computation against an external specification, and the loop has no access to one.

It cannot certify results that are not deterministic, except through a weaker statement. A stochastic step produces a different output every run, so its stored digest will never match a fresh run’s. The verifier can instead record the generator configuration and the seed as inputs, check that they still match, and compare the distribution of fresh runs against the stored one — but that is a statistical statement, not an equality, and it requires the tolerance machinery of Conditioning, Tolerances, and the Digits a Reproduction Can Demand.

It cannot certify floating-point bit-identity across changed hardware. A step whose output depends on vector width, thread count, or fused multiply-add will produce a digest that differs from the stored one even when the input is identical. For those steps the certification has to accept a tolerance rather than a digest, which means the verifier needs to know which steps are bit-deterministic and which are not. Recording that distinction in the provenance graph — a flag per node, plus the tolerance for the tolerant ones — is what makes the verifier’s answer meaningful instead of an endless list of false mismatches.

Engineering the loop

The loop is a scheduler over the recorded graph plus a comparison step. The graph is a DAG, so a topological order suffices, and the parallelism available is the width of the DAG at each level.

import hashlib, json, pathlib, subprocess, sys

def digest(path):
    h = hashlib.sha256()
    for chunk in iter(lambda: open(path, "rb").read(1 << 20), b""):
        h.update(chunk)
    return h.hexdigest()

def verify(step, store):
    # Recompute inputs first: a stale input invalidates the stored output's meaning.
    for name, want in step["inputs"].items():
        if digest(name) != want:
            return f"input changed: {name}"
    outdir = pathlib.Path(step["work"]) / step["id"]
    outdir.mkdir(exist_ok=True)
    subprocess.run(step["cmd"], cwd=outdir, check=True)
    for name, want in step["outputs"].items():
        if digest(outdir / name) != want:
            return f"output mismatch: {name}"
    return "verified"

graph = json.load(open("graph.json"))
for step in graph:                        # graph is already topologically sorted
    print(step["id"][:12], verify(step, "store"))

The store in the call is a content-addressed directory into which each step writes, keyed by the step’s identity, so that two runs of the same step with the same inputs share storage and the comparison is against a fresh write rather than an in-place overwrite. Overwriting in place is the classic mistake: it destroys the stored output before the comparison can run.

Three properties decide whether the loop is maintainable. Determinism of the harness itself: the comparison code, the graph serialization, and the hashing must be stable across runs, or the verifier generates its own mismatches. Isolation of steps: a step that reads the working directory or the host clock produces different results depending on where and when it runs, so each step must get a clean work directory and a controlled environment. Idempotence of the comparison: re-running the verifier must not change the record, which argues for storing results in a separate location from the record and writing the record only on a change.

Continuous replay and the cost of honesty

Running the verifier on every change turns reproducibility into a property that regresses visibly. A commit that edits a module used by a downstream step produces an input-digest mismatch at that step, and the verifier reports it. The author then either re-records the downstream output with its new provenance — which is a legitimate update, the computation changed — or fixes the edit. What the verifier prevents is the silent drift in which the recorded provenance and the current code diverge without anyone noticing.

The cost is compute and the cost is a change in what counts as done. A result that used to be finished when the file was written is now finished when the file is written and the verifier reports that the stored output still reconstructs from the stored inputs. The second condition is the reproducibility claim, and the verifier is the instrument that makes it testable.

The honest schedule is a decision rather than a default. A computation whose inputs change rarely and whose steps are cheap can be replayed on every change. A computation with an expensive step — a long simulation, a large model fit — cannot afford full re-execution on every edit, and the verifier degrades to hash checks on the fast steps plus periodic full replay of the expensive ones. The degradation is acceptable only if the interval between full replays is stated, because the strength of the certification decays between replays and the interval is exactly the decay.

That decay is the link to Bit Rot, Link Rot, and Replaying an Analysis Years Later. An archive preserves the bytes; a replay loop preserves the reproducibility of the computation those bytes describe. Neither substitutes for the other. The archive without the loop is a record that has stopped being checked; the loop without the archive is a check that has no durable object to verify against. Together they convert a reproducibility statement from something a reader has to take on faith into something a reader can rerun, and the rerun is the only certification that survives contact with the passage of time.