Hermetic Actions and Undeclared Inputs

A build graph is a claim about causality: these inputs, run through this command, produce that output. Timestamp and hash comparison as described in Timestamp-Based Incrementality and Its Failure Modes can only check the claim against the inputs the graph names. If the command also reads a file nobody declared, the output depends on something outside the graph, and every guarantee built on top of the graph, including incremental correctness, cache validity, and reproducibility, inherits that hole.

Making the claim true by construction means running the command in an environment where undeclared inputs are not reachable. That property is hermeticity, and it is the load-bearing assumption of Remote Execution and Content-Addressed Input Trees and of the shared caches described in Action Cache Keys and Shared Build Caches.

The channels that leak

Filesystem access is the obvious one. An action that reads an absolute path outside its declared inputs, or that walks upward looking for a configuration file, or that includes a header from a system directory, has an input the graph does not know about. On a typical Linux host the header search path alone reaches several directories populated by whichever distribution and compiler package happen to be installed.

The process environment is the second channel. Any variable can change behaviour: locale variables alter collation and character classification, a timezone variable alters formatted times, variables consumed by a linker or by a language runtime alter output without appearing anywhere in the build file. Because the build tool normally inherits the environment of whoever invoked it, the same action can produce different results for two developers on the same machine.

The third channel is the clock, and it is the one that most reliably breaks caching. An action that stamps a build time, a serial number, or a random identifier into its output produces different bytes on every run, so the digest computed for the output differs every time and no downstream action can be skipped. The fourth is the network: an action that downloads a dependency at build time has an input that can change without any local file changing, and whose value is not recorded anywhere in the graph.

A fifth, subtler channel is the shape of the filesystem itself. Temporary files created with a randomised name, directory iteration order, and the set of environment-provided paths used for tool lookup all vary between machines. Where those values are embedded in output, the divergence is invisible until two machines are compared.

Sandboxing the action

The general mechanism is to give the action a deliberate, minimal view of the system: a private filesystem namespace containing the declared inputs and the toolchain, a private process namespace so it cannot observe or signal unrelated processes, an environment built from scratch rather than inherited, and usually no network at all. On Linux the primitives are namespaces, created with the clone or unshare system calls; a mount namespace combined with a pivot of the root filesystem produces an isolated view, and a user namespace allows that to happen without root privileges.

The heaviest approach is one container per action. Container startup costs time on the order of milliseconds and requires an image to contain the toolchain, which pushes the build system toward shipping tools as part of the input tree. Lighter approaches abandon the container concept and build exactly the directory tree the action needs each time, often with symlinks to content-addressed blobs, so the cost is the cost of creating a handful of directories and links. bubblewrap is a widely used helper of the light kind: it sets up namespaces and mounts for a single command without a daemon or an image.

The read/write split matters for correctness. Inputs should be mounted read-only, so an action that overwrites one of its own inputs fails immediately rather than corrupting a shared artefact that another action is about to read. Outputs should be written to a location that is collected and hashed after the action exits, which is also what makes early cutoff possible.

There is no free isolation on operating systems that lack these primitives; the portable fallback is to construct a temporary directory tree and run the command with a modified working directory and a scrubbed environment. That closes the environment channel and most of the filesystem channel, but a determined command can still reach absolute paths. Where platform isolation is unavailable, the honest position is that hermeticity is approximated, and the cache must be treated as a performance optimisation rather than a correctness guarantee.

Finding the undeclared inputs

Knowing that an action is non-hermetic is the harder half of the problem, because the failure is silent by default. Two classes of technique work in practice.

The first is to make the environment so restricted that the undeclared access fails. Run the action with an environment containing only the variables the build system chose to set, with no entries on any search path that the graph did not supply, and with the working directory as the only readable location other than the toolchain. Under those conditions an undeclared dependency usually turns into a missing-file error naming the path that was reached for. That error is the report of a missing edge.

The second is to record the accesses rather than to prevent them. Running the action under a syscall tracer shows every file opened, and comparing that list against the declared inputs yields exactly the set of undeclared reads.

# Record file opens in the action and report any path outside the declared set.
# -f follows children, -e traces only open-family calls, -o writes the trace.
strace -f -e trace=open,openat,stat,statx -o action.trace ./compile.sh unit.c
awk -F'"' '/open|stat/ {print $2}' action.trace | sort -u > observed.txt
comm -23 observed.txt declared-inputs.txt

The output of an unsandboxed build and a sandboxed build can also be compared. If the sandboxed run fails, the diff between the two executions is a map of what the action was silently reaching for. This is the same comparison that detects non-determinism, and it is the practical way to discover that a build has been depending on the developer’s home directory.

What hermeticity costs

Enforcing isolation changes what a build file may assume, and each change has a price.

Host tools stop being available. Anything the action invokes must be an input, which means the compiler, the assembler, the linker, the shell, and every script they call must be declared and shipped. That is a real amount of storage: a complete C++ toolchain is hundreds of megabytes. The benefit is that the toolchain’s version becomes part of the action key, so a compiler upgrade invalidates exactly the actions it affects instead of requiring a clean rebuild to be safe.

Actions must be explicit about configuration. A tool that used to read a file from a well-known location must instead receive it as an argument or as an input path. This often means wrapping or patching third-party build scripts, and wrappers must themselves be inputs, hashed so that changing a wrapper invalidates the actions that used it.

Some actions legitimately need the clock, and the usual answer is a single declared epoch value that the build system supplies and every tool consults, so that the clock becomes an input with a known value. The convention used across distributions and language toolchains for this is a source-date environment variable, and tools that honour it substitute the declared value anywhere they would otherwise embed the current time.

Actions that resist isolation are the remaining cost. Some steps legitimately need the network, to fetch a dependency from a registry or to sign an artefact with a key held outside the sandbox. Allowing those to run unsandboxed while everything else is confined reintroduces exactly the leak the confinement was meant to close, so the choice is either to move the network access out of the build into a declared step whose result is then consumed by digest, or to accept that such actions cannot be shared across machines at all. Dependency resolution that decides versions at build time belongs to the first category and is usually replaced by a fetch step whose output is pinned; signing belongs to the second, and is better performed on the finished artefact than inside the action that produces it.

The final cost is diagnostic. Failures inside a sandbox are harder to reproduce by hand, and an action that worked before may fail because a file it was implicitly reading is now absent, with the error message pointing at the file rather than at the missing declaration. Tooling that can replay the exact sandboxed command is worth more than it appears, which is one of the reasons a reproducible command representation is valuable well beyond debugging.