Reproducing Training Runs from Recorded Metadata

A trained model that cannot be rebuilt is not an engineering artifact. It is a result. When the model’s behaviour is questioned months later, the questions are always the same shape: which code, which data, which hyperparameters, and does the same run reproduce the same weights. The first three are bookkeeping. The fourth is a property of the software stack the run executed on, and it is where reproducibility claims usually break, because floating-point arithmetic under parallel accumulation is not a deterministic function of its inputs unless the reduction order is.

The record

The record for a training run has to be complete enough that a fresh machine can re-execute it. In practice that means capturing six things, and each is routinely omitted by a system that “tracks experiments” at the level of loss curves.

Source identity. The commit of the training code, and the identity of every dependency that can affect numerics: the numerical library, the accelerator vendor’s kernels, the compiled extension versions. Dependencies matter more than commits because a patch release of a numerical library can change the algorithm behind a reduction and thereby change the last digits of every trained weight.

Data identity. Snapshot identifiers for every input, as described in feature definitions as versioned artifacts. A query string is not a data identity, the same query against a table that has since been backfilled returns different rows.

The split. Which rows went to training, which to validation. Most systems record the split rule but not the resulting assignment, which is insufficient: a rule such as “random eighty percent” applied with a different library version, or with the rows arriving in a different order, produces a different assignment. The seed used for the split and the ordering of the input must both be recorded, or the split must be materialized and stored.

Hyperparameters and seeds. Every value that influences optimization, including defaults that the code did not set. Recording only the values passed explicitly misses the case where a library changed its default between versions.

The environment. Operating system, accelerator model, and driver version. These determine which kernels execute, and kernels differ in reduction order.

The output. Weights, checkpoints, and the evaluation metrics computed from them, stored by content hash so that two runs with the same hash are known to be identical without being compared.

Where nondeterminism enters

With a perfect record, re-execution on the same hardware with the same libraries still may not produce bit-identical weights. Three mechanisms are responsible.

Reduction order. A sum over a batch distributed across parallel units is combined pairwise in an order determined by the parallel schedule, and floating-point addition is not associative. Two runs with the same inputs can sum the same values in different groupings and obtain results differing in the last bit, which after thousands of steps becomes a visibly different model. Fixing the order fixes the result; leaving it to a work-stealing scheduler leaves the result to timing.

Atomic operations in the accumulation of gradients. When many workers update the same buffer without an ordering discipline, the effective summation order depends on which worker arrives first. Some frameworks expose a deterministic mode that forces a fixed order at a cost in throughput; the cost is the price of reproducibility and it is usually paid only for runs that will be compared.

Non-deterministic kernels. A kernel may choose its algorithm from several candidates based on available memory or on a runtime autotuning step, and the choice is made at first invocation. Two runs on the same machine can pick differently if the autotuning cache is cold on one of them. Disabling autotuning or pinning the choice removes the variation.

Randomness that is not captured by a single seed. Data augmentation, dropout, and initialization may draw from several independent generators. A run is reproducible only if each generator’s seed is recorded and the sequence of draws is fixed by the data order. Shuffling the data changes the sequence even with an identical seed, which is why the split and the input ordering belong in the record together.

Recording what actually ran

A training job should write the record as a byproduct, not as a manual step. The reliable pattern is an environment capture at process start, a data snapshot resolution at the moment the first batch is read, and a hash of the resulting record written into the model artifact’s metadata. Manual records diverge from reality at exactly the moment it matters: the run that was launched from a modified working tree, or that loaded a cached dataset from a path that no longer matches its declared source.

The record also has a length. A dependency graph of forty packages has forty versions, and the canonical form of that set needs an ordering convention or the hash differs between two identical environments. Sorting by package name and normalizing version strings is sufficient; the goal is a value that is equal when the environments are numerically identical and different otherwise.

Verification rather than assertion

The claim “this run is reproducible” becomes meaningful when it is a test. The standard shape is a periodic re-execution: take a sample of historical runs, re-execute them from their records on the current infrastructure, and compare. Two levels of comparison are distinguishable.

Metric equality means the re-executed run reaches the same evaluation metric to within a stated tolerance. This is achievable without determinism and catches most serious breakage: if a dependency change altered the algorithm, the metric moves by more than tolerance, and the difference is detected. The tolerance has to be tight enough to be meaningful and loose enough to survive reduction-order variation, and choosing it is a modelling judgement rather than a technical one.

Bitwise equality of weights means the run is deterministic under its recorded environment. This is much stronger, requires the determinism switches described above, and is breakable by an operating system update that changes a threading library. Its practical value is in debugging: when a training bug is suspected, a bitwise-reproducible setup allows the bug to be isolated by bisecting the run rather than by reasoning about it.

The two levels serve different purposes and it is a mistake to insist on the second as a general requirement, because the cost of deterministic execution is paid on every run rather than on the ones being investigated.

Artifacts that outlive their environment

The record is only useful while the environment it names is reconstructible. Package indexes lose versions, base images disappear, accelerator drivers are withdrawn. The measures that survive are container images pinned by digest rather than by tag, dependency sets resolved to exact versions and stored as lockfiles rather than re-resolved at build time, and model artifacts stored with enough information to run inference without rebuilding the training stack. The model binary and its preprocessing, packaged together, outlive the training environment by years, and the record’s role after that point is to answer questions rather than to re-execute.

What remains unfixable is the case where the recorded data itself no longer exists, which is why the retention policy of source tables is a reproducibility property, as noted in point-in-time correctness in training set assembly. A record that names a snapshot the warehouse has since deleted documents the run without permitting its repetition, and that limitation should be recorded explicitly rather than discovered when someone attempts the rebuild.