Hermetic Builds: Isolating the Network, the Clock, and the Filesystem

A build is hermetic when its output is a function of its declared inputs and nothing else. The phrase “and nothing else” is where the difficulty lives, because an ordinary build process has access to a very large amount of ambient state: every file on the machine, the entire network, the clock, the process table, the environment, and the identity of the user running it. Any of that can influence the output. A build that silently depends on one of those channels will still succeed tomorrow and fail next week, and — more importantly for supply chain purposes — a build that can reach the network can pull in a dependency that no one declared and no one will see in the dependency list.

The channels that have to be closed

The network. A build step that downloads at compile time makes the artifact depend on whatever the remote server served at that moment. This is not hypothetical: many build systems fetch dependencies lazily, and many build scripts invoke a package manager in the middle of the build. Closing the network channel requires that every input be materialized before the build starts, from a record of what was resolved, and that the build then have no route to fetch anything. The record is the lockfile, and the fetch step is the one place where the artifact hash from what a SHA-512 integrity hash proves gets checked.

The clock. Wall-clock time reaches the output through macros, generated version strings, and archive timestamps. A hermetic build supplies a fixed time or makes the time source fail, and the tooling normalizations that keep the fixed time out of the artifact are the subject of reproducible builds and the sources of nondeterminism.

The filesystem outside the declared inputs. A compiler that can read /usr/lib or a home directory can pick up a header that was not in the declared set. It can also pick up a different version of a tool. Isolation means the build sees a filesystem assembled from exactly the declared inputs plus the declared toolchain, with no host paths visible.

The environment. Variables are ordinary inputs that are easy to forget. A build that reads HOME, PATH, or a language-specific variable inherits whatever the invoking shell had. A hermetic build passes an explicit environment and clears the rest.

Process and resource state. The number of CPUs visible, the memory limit, and the identity of the invoking user can all change output through parallel scheduling decisions or through conditionals in scripts. Setting them explicitly makes them part of the declared input rather than a hidden one.

Randomness. Any call to a random source makes an output nondeterministic unless the seed is fixed or the result is excluded from the artifact. That requirement, and the full taxonomy of the leaks that isolation alone cannot remove from the bytes, belongs to reproducible builds and the sources of nondeterminism; isolation decides what a build can see, while normalization decides what the resulting bytes contain.

Mechanisms, from coarse to fine

The practical implementation of isolation ranges from “run the build in a container with networking disabled” to per-action sandboxing with syscall filtering, and the difference is the granularity of what is visible.

At the coarse end, a container with a private network namespace and no interfaces gives a build no route off the machine, which closes the network channel entirely and fails loudly the first time a build step tries to fetch. A read-only root filesystem with a writable scratch directory closes the accidental-write channel. A fixed SOURCE_DATE_EPOCH closes the clock for tools that honor it. This level is achievable with ordinary container tooling and catches the majority of accidental non-hermeticity.

At the fine end, each build action is sandboxed separately and its declared inputs are materialized into a private mount namespace. The dependency graph is then the enforcement: an action that reads a file not listed as an input fails rather than silently succeeding. Systems that implement this model generally derive the input list from the build description and verify it during execution, which is why such builds are described as requiring the full dependency graph to be declared up front. The relative paths, mount points, and working directory inside the sandbox are fixed by the sandbox, which also removes the absolute-path leak discussed in the nondeterminism article.

Linux provides the primitives: user namespaces for unprivileged mount and PID isolation, mount namespaces for the private filesystem view, network namespaces for the absent route, and seccomp filters to remove syscalls an action has no legitimate use for. A sandbox built from these pieces has to handle the case where a compiler legitimately forks, spawns child processes, and maps executable memory, so the filter is necessarily permissive at the syscall layer; the tighter control is the filesystem view.

# Minimal hermetic execution: empty network namespace, private mount view,
# fixed time and environment, read-only toolchain.
unshare --user --map-root-user --net --mount --pid --fork \
  --mount-proc \
  env -i SOURCE_DATE_EPOCH=1700000000 PATH=/toolchain/bin \
  bwrap_shim --ro-bind /toolchain /toolchain \
             --bind "$PWD/src" /src --chdir /src \
             --tmpfs /tmp --unshare-all \
  make -j1

The point of the command is the flags rather than the specific wrapper: --net removes the route, env -i removes the inherited environment, SOURCE_DATE_EPOCH and -j1 remove the clock and scheduling variation, and the bind mounts define the visible filesystem. -j1 is a deliberate choice for the first hermetic run: a parallel build can mask an ordering dependency that will surface later on a machine with a different core count, so the parallel setting is itself an input that must be fixed.

What isolation does not do

Isolation constrains what a build can see; it does not constrain what a build is allowed to do to its declared inputs. A malicious dependency that runs during the build can modify the source tree, write to the output directory, or exfiltrate through the only channels left: the output itself, the exit status, and timing. The output channel is real. A build action that encodes information in the artifact it produces is not detected by any filesystem or network control, which is why isolation is paired with the requirement that the output be reproducible and reviewed — the artifact is checked against a rebuild, not merely against a policy.

Isolation also does not resolve the toolchain question. The compiler is inside the sandbox, and the sandbox proves it is the declared compiler only if the toolchain input is itself verified by digest. A pinned path to a mutable toolchain directory is not a pinned toolchain.

Finally, isolation costs time and flexibility. Pre-fetching every input makes the metadata fetch a separate, serialized step. Removing network access breaks build scripts that were written to fetch, often as a convenience, and the migration work is per-project. Caching across isolated actions requires a cache keyed on the full input set, which is only computable when the inputs are declared — the same declaration that isolation enforces.

Two-party verification and the trust it removes

The strongest use of hermetic builds is not isolation for its own sake but the elimination of the builder as a trusted party. If a build is hermetic and reproducible, then two independent parties who start from the same source and toolchain produce the same digest, and either party’s result can be checked against the other’s. A consumer no longer has to accept that the artifact shipped by a vendor is what the vendor’s source produces; the consumer or a third party can produce the same bytes and compare.

This is where the pieces join. The source is pinned by commit, the toolchain by digest, the dependencies by lockfile and artifact hash, the environment by the sandbox, and the result by digest. What remains is the question of which source commit was supposed to be built — and that is answered by provenance, in the form described in provenance attestations and the digest they bind, where the isolation level of the builder is precisely what distinguishes the higher build levels from the lower ones.

The cost of two-party verification is that both parties must maintain the capability. Distributions that rebuild every package independently, and projects that arrange for a second builder outside their own infrastructure, are the deployments that actually achieve it; a single-party “reproducible” build reduces the risk of accidental drift but does not change who is trusted.

The remaining gap after isolation and cross-checking is the semantic one: the source may be intact and the build faithful, and the software may still be vulnerable or malicious by design. Isolation is about provenance of bytes, and is silent on the behaviour of those bytes.