Layer Cache Keys and What Makes a Build Reproducible

A layer digest is a hash of compressed tar bytes. That sentence contains the whole problem. The tar bytes include file modification times, ownership, permission bits, and the order in which entries appear. The compressed bytes include the compressor’s choices about block sizes and match distances. Neither is determined by file content alone, so two builds that produce identical files can produce different layers, different digests, and therefore different images. Everything about build reproducibility follows from closing that gap, and everything about build caching follows from the fact that the gap is not always closable.

The cache key is the instruction plus its inputs

A build executes instructions in order, and each instruction produces a layer. The cache lookup for an instruction typically keys on a hash of the instruction text, the identity of the base image, and (for instructions that copy files in) a checksum of the files being copied. The result is a mapping from that composite key to a layer digest, held either in a local store or in a registry that stores cache manifests.

The consequence is that instruction text is part of the cache key, byte for byte. Reordering two arguments, changing a comment that is retained in the instruction string, or altering whitespace inside a shell command all invalidate the entry. This is why cache-friendly images put rarely changing instructions early: the key chain is a prefix chain, and an invalidation at position n discards the entries for every instruction after n, even if those instructions and their inputs are unchanged.

Consider the difference between two orderings of the same dependency installation.

# Order A: manifests first, sources second.
COPY package.json package-lock.json /app/
RUN npm ci
COPY src/ /app/src/

# Order B: everything at once.
COPY . /app/
RUN npm ci

In order A, the npm ci layer’s key depends on the two manifest files, which change rarely. Editing a source file invalidates only the final copy. In order B, the npm ci key depends on the checksum of every file in the build context, so any source edit invalidates the dependency installation and re-downloads the package tree. The images are functionally identical; the cache behaviour is not. This is the single largest structural difference between image builds that take seconds and image builds that take minutes, and it is a property of instruction ordering rather than of any tool.

The same reasoning explains why build contexts should exclude what the build does not need. A context that includes a local dependency directory, a test fixture tree, or a version control directory inflates the checksum input for COPY . and makes every unrelated change a cache miss. Exclusions are specified per-build, and the default is usually to include everything not explicitly excluded, which means a large repository silently pays for a large context on every build.

Diff IDs, layer digests, and the two-level identity

The image config records a diff_id per layer: the digest of the uncompressed tar stream. The manifest records the digest of the compressed blob. These are two different hashes over two different byte sequences, and the relationship between them is what allows the same logical layer to be stored with different compression.

For reproducibility, the diff ID is the more interesting value, because it is independent of the compressor. Two builds that produce byte-identical tar streams have identical diff IDs regardless of whether one used gzip level 6 and the other zstd. They will still have different layer digests and different manifests, so the image digest differs, but the filesystem content is provably the same.

For caching, the layer digest is the value that matters, because it is what the registry stores and what a subsequent build can pull instead of recomputing. A cache that keys on diff IDs can share work between builds with different compression settings; a cache that keys on blob digests cannot. Most implementations do the latter, which is a deliberate simplification: compression settings are usually fixed per builder, so the extra sharing would rarely pay for the extra bookkeeping.

Tar is the nondeterminism source

If a build runs the same commands on the same base image with the same inputs, the filesystem it produces is usually identical in content. The tar stream that captures the difference is not. The GNU tar format stores, per entry, the modification time to the second, the uid and gid, the user and group names, the permission bits, and the entry name. Any of these differing produces a different stream.

Timestamps are the most common offender. Any step that writes a file records the current time, so a build that runs apt-get install or compiles a binary produces files whose mtimes differ between runs. A build that copies files from the context carries the mtimes from the host filesystem, which differ between checkouts. Tools that normalize these values exist precisely because the tar format has no notion of a canonical timestamp.

Ownership and permissions are the second offender. Running a build as a different user changes uid and gid on every file created by the build steps. A chmod that is not part of the instruction but is applied by an installer script changes permission bits. Since these values are recorded verbatim, an image built by a rootless builder and one built by a root builder can differ on every layer even when the instruction text is identical.

Entry order is the third. A tar stream is a sequence, and directory traversal order is not specified by POSIX; it depends on the underlying filesystem’s directory index. Two builders unpacking the same base image on different filesystems can enumerate a directory in different orders and emit entries differently. Sorting entries before writing the tar removes this, at the cost of a pass over the metadata.

import tarfile, os

def add_sorted(tar, root, epoch=0):
    entries = []
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames.sort()
        filenames.sort()
        for name in dirnames + filenames:
            full = os.path.join(dirpath, name)
            entries.append(os.path.relpath(full, root))
    for rel in sorted(entries):
        info = tar.gettarinfo(os.path.join(root, rel), arcname=rel)
        # Normalize the fields the tar format records but content does not determine.
        info.mtime = epoch
        info.uid = info.gid = 0
        info.uname = info.gname = ""
        if info.isfile():
            with open(os.path.join(root, rel), "rb") as f:
                tar.addfile(info, f)
        else:
            tar.addfile(info)

Normalizing uid and gid to zero and names to empty is not always correct (a base image that depends on file ownership for a service to run will break) but it is the standard trade: determinism in exchange for the requirement that permissions be expressed explicitly in the build rather than inherited from the builder’s environment.

Compression adds a second layer of variance

Gzip output is a function of the input bytes, the compression level, the deflate implementation, and the header. Different zlib versions and different implementations (zlib, zlib-ng, Go’s compress/flate) produce different streams for the same input at the same nominal level. The gzip header also carries an mtime field by default, which some implementations zero and others populate from the input file’s timestamp.

The result is that a diff ID match does not imply a blob digest match across builders using different compressors. Registries deduplicate on blob digests, so two teams building the same image with different compression settings store two copies of the same content. This is invisible until someone compares image sizes or registry storage and finds an unexplained factor of two.

Rebuilding and comparing is the only real test

The operational check for reproducibility is straightforward: build twice from the same inputs in environments that differ in the ways that matter, and compare the resulting digests. If they match, the build is reproducible with respect to those differences. If they do not, the first mismatching layer localizes the cause, and the usual culprits are in the order above: timestamps first, then ownership, then ordering, then compression.

Comparing layer by layer is more informative than comparing image digests, because the image digest changes if any layer changes and tells nothing about which. Unpacking both layers and diffing the file trees, with metadata included, distinguishes “different content” from “same content, different metadata”. That distinction determines whether the fix is in the build script or in the packaging step.

Reproducibility also has a supply-chain consequence that is easy to miss. A reproducible build is one where an independent party can take the source, rebuild, and confirm the digest. That confirmation is what lets a verifier connect a published provenance statement to bytes it can regenerate itself, rather than to a digest it must take on faith. Without reproducibility, provenance says “this builder claims these inputs produced this output”; with it, the claim is checkable. Verifying Provenance by Rebuilding goes into what a rebuild comparison can and cannot establish.

The cache also has a trust dimension that is separate from its correctness. A cache entry is content addressed by digest, so a poisoned entry can only be served if it matches the digest the build expected, which it will not, unless the attacker can also influence the key. Where the key includes only instruction text and not input checksums, an attacker who can modify the build context changes the key, and the poisoned result is legitimately cached under it. This is the reason remote caches must be scoped to trusted producers, a point developed in Pull-Through Caches and Where the Trust Anchor Sits.