Rebuilding a Pinned Artifact Without Its Original Environment

Storing the bytes is cheaper than storing the ability to produce them, right up to the point where the bytes are gone or the bytes are suspect. Then the only remaining option is to rebuild, and rebuilding depends on an environment that no longer exists: a compiler version that the distribution has removed from its mirrors, a base image that the registry garbage-collected, a dependency whose registry entry was unpublished. A pipeline that intends to rebuild later has to treat the environment as an artifact in its own right.

Two retention strategies and what each costs

Storing output bytes costs storage proportional to the artifact and is exact. The bytes are what ran; nothing about them has to be recovered. The risks are loss and longevity: the storage medium fails, the format becomes unreadable, the encryption key is lost, or the retention policy deletes the object because nothing in the policy’s roots referenced it.

Storing the recipe costs storage proportional to the inputs, which is usually much smaller, and it is not exact. Reconstruction depends on everything the recipe names remaining fetchable, and on the build being deterministic enough that the reconstruction produces the same bytes rather than merely equivalent behaviour.

Neither strategy subsumes the other, and the practical answer is a mixture. Release artifacts are stored as bytes, because the exact binary that a customer runs must be recoverable byte for byte. Debug symbols, source tarballs, and dependency archives are stored as inputs, because they are needed for investigation rather than execution and they compress well. Everything else — the intermediate build outputs, the compiler’s temporary files — is not kept at all.

The mixture has a consistency requirement that is easy to violate: the recipe that is stored must correspond to the bytes that were released. If the release was built from a working tree with a modification applied after the commit was recorded, the stored recipe reconstructs a different artifact, and nobody discovers it until the artifact is lost and the rebuild fails verification.

What the recipe has to name

The list of pinned inputs is short in principle and long in practice, and each entry is a place where the pin can be forgotten.

Source. A commit digest in a content-addressed version control system, not a branch name, and not a tarball whose contents were never verified against the commit. Where the source repository itself may disappear, an archive of the tree is the pin, with its digest recorded.

Toolchain. The compiler and its exact build, named by digest. A version string is not a pin: distribution packages of the same upstream version differ by patches, and a toolchain built locally differs from the distribution’s build. This is the input most often recorded loosely and the one that most reliably causes a rebuild to differ.

Base filesystem. A container image digest for containerized builds, and a package manifest with digests for builds that install packages into a filesystem. Package repositories remove versions, so the pin is only durable if the packages themselves are archived somewhere under the pipeline’s control, or if the repository is contractually immutable.

Dependencies. A lockfile pins versions and, when the lockfile records content digests, pins bytes. A lockfile that records versions and hashes from a registry is only as durable as the registry’s immutability policy; registries that allow owners to republish a version break the pin silently.

Non-input inputs. The things that are not artefacts but still affect the output: the time zone, the locale, the character encoding, the system clock. The clock is handled by fixing a timestamp value and having tools honour it, which is what the SOURCE_DATE_EPOCH convention exists to make uniform across toolchains. The locale and time zone have to be part of the environment the build is given, because a build that reads them has them as inputs.

A recipe that records all of this is a data structure, and its value comes from being machine-readable and checkable:

{
  "source": { "vcs": "git", "revision": "0b1d4f7c8a92e5d3" },
  "toolchain": {
    "compiler": { "name": "clang", "storeDigest": "sha256:6a1f...", "version": "18.1.8" },
    "linker": { "name": "lld", "storeDigest": "sha256:b70c..." }
  },
  "rootfs": { "imageDigest": "sha256:c93a...", "packages": "pkg-manifest.sha256" },
  "environment": { "LANG": "C", "LC_ALL": "C", "TZ": "UTC", "SOURCE_DATE_EPOCH": "1704067200" },
  "flags": ["-O2", "-g", "-fdebug-prefix-map=/build=/build"],
  "outputs": { "app-linux-amd64": "sha256:9f2c..." }
}

The outputs field is what makes the recipe verifiable. A rebuild that produces those digests has reconstructed the artifact; one that does not has reconstructed something else, and the difference is the interesting part.

Reconstructing the environment when the sources are gone

The hard case is a rebuild attempted after a mirror has been cleaned. The specific failures are predictable.

A compiler version is gone from the distribution. The mitigation is to archive the toolchain, which for a compiler means archiving its store closure: the compiler binary, its shared libraries, its standard library headers and runtime, and the loader it needs. A toolchain that depends on a system library version no longer present in the current distribution cannot run from an archive alone; it needs the library too. Systems that name build inputs by content hash, in the manner of store paths that embed a hash of a derivation’s inputs, solve this by construction, because the closure is what the store path transitively names and the store retains whatever is reachable from a live root.

A base image is gone. Registries that garbage-collect by age rather than by reachability will delete an image that no tag points at, and a digest-pinned reference is precisely a reference with no tag. The mitigation is to keep a root that references the digest, which is policy rather than mechanism, and to keep a copy in a second store.

A network resource has changed. A build that downloaded a tool at build time, from a URL that now serves a different version, cannot reproduce. The mitigation is to fetch inputs before the build, verify them by digest, and treat the verified local copies as the build’s inputs. This is also what makes the build hermetic, so the same change serves both purposes.

Certificate and key material has expired. A rebuild that must verify a signature collected at release time needs the certificate chain as it was, a timestamp proving the signature predates expiry, and the trust roots as they were. RFC 3161 defines a time-stamp protocol that exists for this purpose, and systems that need long-lived verification archive the proofs alongside the artifacts rather than re-deriving them. The same archival applies to the provenance records that bind an artifact digest to its builder, which are unverifiable once the keys that signed them have been revoked and their certificates have expired.

What a rebuild can and cannot verify

A successful rebuild that produces the recorded digests is the strongest result available: the recipe is complete with respect to the artifact’s content, and the artifact’s identity is confirmed by recomputation, which is what content addressing means by identity. This is what the reproducibility project’s two-party verification does in spirit, and it is the only check that detects a tampered artifact whose storage was compromised in a way that preserved the digest record.

A rebuild that produces different digests has several possible explanations, and distinguishing them is the work. Remaining nondeterminism accounts for differences in a few fields, usually timestamps, paths, or ordering; the diagnostics are the same ones used to remove nondeterminism in the first place, described in Sources of Nondeterminism in Build Output. An incomplete recipe accounts for larger differences, and the differing output usually points at which input was missing. An actual mismatch in the source or toolchain means the rebuild is not a rebuild.

For cases where bit-identical reconstruction is not achievable, the fallback is a documented equivalence: the two artifacts are compared for behaviour, with the differences enumerated and explained. This is weaker evidence, and the explanation is what carries the weight. A difference attributed to a build path change with everything else identical is credible; a difference that nobody has explained is not, and the honest classification of such a case is that the artifact has not been verified.

Retention policy as reachability

Retention is a graph problem disguised as a date problem. Artifacts are kept because something references them: a release record, a running deployment, a support agreement, a legal hold. Deleting by age flattens that structure and removes objects that are still referenced.

The workable design marks and sweeps. Roots are the release records and deployments that must survive; the sweep walks from the roots through the recipe to the inputs and keeps everything reachable, with a grace period so that in-flight builds do not have their inputs deleted underneath them. The graph is deeper than it looks, because a stored recipe points at a toolchain, a toolchain points at its libraries, and so on; a policy that keeps recipes and deletes toolchains has kept the map and destroyed the territory.

The cost is dominated by the sources-available question. Keeping one copy of a compiler is small; keeping ten years of compilers grows monotonically, because old entries are rarely deleted by the same rules that delete artifacts. Framing the policy in terms of what must be recoverable — which releases, for how long, under which agreements — produces a bounded set of roots, and the storage follows from the graph. The alternative, keeping everything, fails not because the storage is unaffordable but because nobody can then say what is still recoverable, and an archival system whose contents are unknown cannot be relied on to have the one artifact that matters.