Reproducibility, Replicability, and the Boundaries Between Them

A computational result can be re-derived, re-obtained, and re-believed, and those are three different achievements with three different failure modes. Conflating them produces arguments that sound conclusive and prove nothing: an analysis that re-runs on the original data says nothing about whether a different dataset would give the same answer, and an independent replication that agrees says nothing about whether the original pipeline was even executable.

The National Academies report on the subject draws the distinction sharply, and its terminology is worth adopting because the words are otherwise used interchangeably in practice and the ambiguity hides real disagreements.

Three claims hiding behind one word

Reproducibility is the claim that a specified computation, given the same inputs, environment, and analysis code, produces results consistent with those previously obtained. The scope is deliberately narrow: the data is fixed, the code is fixed, and what is being tested is whether the machinery can be operated again and whether the reported numbers follow from it. A reproduction that fails identifies a defect in the record — missing files, undocumented manual steps, unstated parameters — not necessarily an error in the science.

Replicability is the claim that a new study, using new data, reaches a conclusion consistent with the original. The data collection may be repeated with the same protocol, or a fresh sample may be drawn. Here the machinery is assumed to work and the question is whether the finding is a property of the world rather than of the particular sample, instrument, or analyst.

Computational reproducibility is the special case where everything between the input data and the reported numbers is software. It is the only one of the three that can be settled by mechanical means, and it is the only one that is fully under the control of the people who wrote the analysis. That asymmetry explains why it attracts so much engineering effort and why it is often mistaken for the whole of the problem.

A fourth term, repeatability, is sometimes used for the narrowest case: the same analyst, same instrument, same day, getting the same measurement. Metrology treats it as the floor of a nested hierarchy, with reproducibility across laboratories as the broader claim. The general pattern holds at every level: the tighter the scope, the more a failure points at recording and the less it points at the phenomenon.

What has to be pinned for a computational reproduction

Enumerating the requirements makes the engineering concrete, because each item is a separate artifact that can be missing independently.

The input data must be identified by content rather than by name. A path is not an identity: data.csv can be regenerated, truncated, or silently re-exported with a changed column order, and the analysis will run to completion on the wrong bytes. A cryptographic hash over the file contents gives an identity, and recording it at read time gives a check.

The code must be the exact revision that ran, together with any generated code, templates, and notebooks. Version control gives this only if the working tree was clean at execution time; uncommitted edits are invisible in a commit hash.

The environment must be reconstructable: interpreter build, package versions, system libraries, compiler and its flags for compiled extensions, and the numerical libraries those extensions link against. Two builds of the same package version can differ in floating-point results when they dispatch to different kernels or were compiled with different vectorization settings.

The parameters and seeds must be explicit. Every pseudorandom generator needs its seed and its algorithm recorded, because default seeding from entropy makes a run irreproducible by construction, and because different generators consume their state differently.

The execution order must be fixed independently of any interactive session, for the reasons described in Execution Counts and the Hidden State of a Notebook Kernel.

The hardware class matters for a narrower set of results: anything depending on floating-point reduction order across threads, on GPU kernel implementations, or on the number of available cores when the parallelism changes summation order. Recording a hardware description is cheap; knowing which results actually depend on it is not.

Each item is independently omittable, and each omission produces a failure that looks like a different kind of problem. A missing seed produces a result that differs in every digit. A missing package pin produces a result that differs in the last few digits and gets attributed to noise. A missing input hash produces a result that is internally consistent and wrong.

Why agreement is a tolerance question, not an equality question

Floating-point arithmetic makes bitwise identity an unreasonable target for most results, even when the same code runs on the same machine. The order of reductions, the use of fused multiply-add, the presence of extended-precision intermediate registers on some architectures, and the vector width chosen at runtime by a library all change the low bits.

A reproduction therefore has to state a tolerance, and the tolerance has to be justified by the quantity being reported. For an iterative solver that terminates when the residual norm falls below a threshold, a relative difference of a few units of roundoff is unsurprising; for an integer count of events in a discrete simulation, any difference is a genuine discrepancy. Reporting a result to more significant digits than the computation’s conditioning supports creates an apparent irreproducibility that is really an overstatement of precision; the propagation of conditioning into reported digits is the subject of Condition Numbers and the Cost of Reporting Too Many Digits.

The asymmetry matters for adjudication. A reproduction that lands within a stated tolerance supports the claim that the reported numbers follow from the recorded inputs. It does not support the claim that the numbers are correct, that the method is appropriate, or that the conclusion is sound. Those are separate judgments, and the mechanical check is silent on all of them.

What a replication tests that a reproduction cannot

A replication introduces variation the original analysis never saw. New data from a new sample, a new instrument, or a new site. If the original result depended on an idiosyncrasy of its dataset — a particular distribution of missing values, an unusual cluster of observations, a batch effect in one laboratory’s assay — the replication is where that dependence surfaces.

This is why replication cannot be reduced to running the original pipeline on fresh inputs. It requires the pipeline to be a faithful implementation of the stated method, which is a claim about the code matching the paper, and it requires the fresh data to be comparable, which is a claim about measurement. Both are contestable in ways that a reproduction is not.

The statistical framing follows from that. A replication is a test of whether the effect survives resampling, and its outcome is a statement about evidence, expressed in the same terms as the original inference. Agreement between two studies, each with its own uncertainty, is weaker evidence than the product of their precisions suggests, because shared methodological assumptions produce correlated errors. Two implementations of the same flawed estimator will agree with each other while both being wrong.

Failure modes that get misdiagnosed

Four patterns account for most confusion in practice.

A reproduction failure treated as a scientific refutation. A paper’s analysis cannot be re-run because a dependency is gone, so the result is described as unreproducible, and the description is then read as a challenge to the finding. The failure is in the record, and the finding may be entirely sound.

A successful reproduction treated as validation. The pipeline runs and emits the published table. This establishes that the table came from that code and data. It establishes nothing about whether the table means what the paper says it means.

A failed replication treated as misconduct. A replication that disagrees may indicate an error, or may indicate that the replication used a different population, instrument, or analysis choice. Distinguishing these requires the replication to be documented as carefully as the original, which is the same requirement, applied twice.

A replication that agrees treated as confirmation of the mechanism. Agreement is consistent with the original mechanism and with any other mechanism that predicts the same observable.

The last is the hardest to see, because it is not a failure of evidence but a gap between the evidence gathered and the claim made. It is the reason that records of what was computed, which is what Content-Addressed Provenance Graphs for Derived Outputs provides, do not substitute for records of why each step was believed appropriate.

Engineering consequences

The distinction has a practical payoff: it tells you which infrastructure actually improves which claim.

Container images, lockfiles, and pinned interpreters move the reproducibility claim from unsupported to supported. They do essentially nothing for replicability, because replicability depends on data collection and statistical design, which no build system can constrain. Workflow engines that re-execute a dependency graph when inputs change help reproducibility and, in a limited way, replicability, by making it cheap to run the same analysis on a new dataset — but the new dataset still has to be comparable, and that is a judgment about measurement.

Conversely, statistical tools — power analysis, preregistration, held-out samples — address replicability and leave reproducibility untouched. A preregistered analysis plan that cannot be executed is not evidence of anything.

The two claims are complementary, not nested, and a record that supports one is not automatically a record that supports the other. The most common mistake in computational practice is treating the achievable one as if it covered both, and then reporting a successful re-run as if it were a confirmed finding.