Content-Addressed Artifact Identity and the Cost of Mutable Tags

A pipeline that publishes an artifact as service:latest and deploys by that name is not deploying an artifact. It is deploying whatever the registry happened to be serving at the moment each consumer asked. Two hosts that pull a minute apart can receive different bytes, and nothing in either host’s configuration records which bytes it received. The name is a query, not an identity.

A digest names bytes, and only bytes

An artifact digest is the output of a cryptographic hash function applied to the artifact’s content. FIPS 180-4 specifies SHA-256 as producing a 256-bit value, conventionally written as 64 hexadecimal characters. Two properties matter for build pipelines and they are separate properties:

Collision resistance means it is computationally infeasible to find two distinct byte strings with the same digest. This is what lets a 64-character string stand in for a multi-gigabyte file. Preimage resistance means that given a digest, recovering an input that produces it is infeasible. This is what prevents someone from constructing an artifact to match a digest that a deployment record already names.

Both properties are about the function, not the artifact. The digest of a file says nothing about who produced it, when, from which source, or with which compiler. It is a pure function of the bytes and is therefore identical for every copy of those bytes anywhere. That is the whole value: identity that survives copying, mirroring, and re-upload.

Hashing a tree without reading it twice

A single file is easy to hash. A directory tree or a container image is not, because the artifact is a structure, not a stream, and the naive approach hashes the whole thing as one blob. That works, but it forces every consumer to read every byte before it can verify anything, and it makes partial transfer useless: one changed file invalidates the digest of everything.

The Merkle construction fixes this. Each file is hashed individually; the digests are then combined into a parent digest, which is combined with its siblings, up the tree. A change to one leaf changes every digest on the path from that leaf to the root and nothing else. A consumer holding the root digest can verify any subtree it has fetched, and can fetch only the subtrees whose digests differ from what it already has.

Container images use this shape directly. The OCI image specification describes a manifest that lists a configuration object and a set of layers, each referenced by a descriptor containing its media type, its size, and its digest. The manifest itself is stored under its own digest, so a tag resolves to a manifest digest, which names a fixed set of layer digests, each of which names fixed bytes. Layers are content-addressed individually, which is why two images sharing a base layer transfer that layer once.

The same idea applies to source trees. Hashing a directory as one blob makes every commit produce an unrelated digest. Hashing a sorted list of (path, mode, content-digest) tuples produces a digest that changes only when the tree changes, and it can be computed incrementally because unchanged files keep their leaf digests.

Tags are mutable pointers, and resolution is a separate step

A registry tag is a name bound to a digest, and the binding is mutable by design. latest exists so that a human can ask for the most recent push; it is reassigned on every push that targets it. Nothing about the storage layer prevents a tag from being reassigned to a completely unrelated artifact, and most registries offer an operation to do exactly that.

The failure this creates is a time-of-check to time-of-use gap. A deployment process resolves the tag to a digest, then pulls the manifest and layers. Between those two operations the tag can move. Worse, the pull may be spread across many hosts over minutes or hours, so different hosts resolve at different times and each gets a self-consistent but mutually different image. The deployment is then a mixture of two builds, and no per-host record distinguishes them unless the digest was captured at pull time.

Tags also destroy rollback precision. “Roll back to the previous version” is well-defined only if the previous version has an immutable name. If the previous deployment was recorded as a tag, and that tag has since moved, the rollback redeploys the newer artifact and reports success. The failure is silent and appears only under investigation.

Digest-pinned references close the gap. A deployment descriptor that names a manifest digest cannot resolve to different bytes on different hosts, because the digest is verified against the content on arrival. The registry can still serve the content or refuse; it cannot serve something else.

Every input has to be named the same way

Pinning the output is only half the problem. If the build that produced the output took a floating input, the digest is stable but the recipe that produced it is not reproducible. A build is a function of its inputs, so the identity of the build should be a function of the identity of its inputs.

Nix makes this structural: a store path incorporates a hash over the derivation’s inputs, so the path name changes whenever any input changes, and a path that exists already has exactly the content it names. The inputs that must be covered are broader than source code: the compiler and its version, the base image or root filesystem, the dependency lockfile contents, the build script itself, and the environment variables the script reads.

The hole in any such scheme is the input nobody hashed. A build that reads the system clock, the hostname, a network service, or an unlisted file in the working directory has an input outside the digest. The digest still identifies the bytes that came out, but it no longer tells you what would come out if you ran the build again, and it cannot tell you whether a cached artifact is valid for a given source revision. That is a reproducibility problem, not an identity problem, and it is covered in Sources of Nondeterminism in Build Output.

A minimal canonical digest over a source tree, ignoring the metadata that does not affect compilation:

import hashlib, os

def tree_digest(root: str) -> str:
    entries = []
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames.sort()
        for name in sorted(filenames):
            full = os.path.join(dirpath, name)
            rel = os.path.relpath(full, root).replace(os.sep, '/')
            st = os.stat(full)
            if not os.path.isfile(full) or os.path.islink(full):
                continue
            h = hashlib.sha256()
            with open(full, 'rb') as f:
                for chunk in iter(lambda: f.read(1 << 20), b''):
                    h.update(chunk)
            # mode is included; mtime, uid, gid and owner names are not,
            # because a checkout reproduces mode but not timestamps.
            entries.append(f'{rel}\0{st.st_mode & 0o111:o}\0{h.hexdigest()}')
    outer = hashlib.sha256()
    for line in sorted(entries):
        outer.update(line.encode() + b'\n')
    return outer.hexdigest()

The ordering and the separator are load-bearing. os.walk yields directory entries in filesystem order, which differs between machines, so the sort is required. The NUL separator prevents a path containing a newline from colliding with a digest line. Without both, the same tree can produce two digests on two machines.

Where the digest stops being enough

Content addressing is not a complete answer to artifact identity, and the gaps are worth naming because each has produced a real class of bug.

Identical bytes from different builds collapse to one name. That is the point of content addressing, but it means the digest cannot distinguish a build produced from a reviewed commit from a build produced from a working tree with local edits. If provenance matters, it must be recorded separately, as attested metadata bound to the digest rather than inferred from it, which is the subject of Artifact Provenance from Commit to Deployed Digest.

Hash algorithm agility is a second gap. SHA-256 is the current default in the ecosystems that matter, but artifacts hashed under an older algorithm remain in circulation, and a registry must be able to accept both while a client asks for one. The OCI specification handles this by allowing a digest to carry its algorithm as a prefix, so sha256: and a future algorithm can coexist in the same field without ambiguity. A pipeline that stores bare hex digests without the algorithm cannot migrate, because it has no way to say which function produced the string.

Third, the digest is a verification mechanism, not an authorization one. Anyone can compute the digest of anything. A deployment that checks “the bytes match the digest I was given” has verified integrity relative to that digest and nothing else; whether the digest came from a trusted source is a separate question. The mirror-image problem, where an untrusted build writes into a cache that trusted builds read from, is treated in Trust Boundaries in Shared Build Caches.

Finally, content addressing interacts badly with artifact formats that are not canonical. A tar archive stores modification times, ownership, and entry order in its headers, so the same files archived by two tools produce different bytes and therefore different digests. The fix is to normalize the archive before hashing — fixed timestamps, numeric ownership, sorted entries — which is the same discipline that makes the build itself reproducible. The SOURCE_DATE_EPOCH specification exists because this normalization has to be agreed on across tools rather than invented per project.

The practical rule that follows is narrow and useful: name artifacts by digest everywhere a machine makes a decision, keep tags for humans, and record the resolved digest alongside every deployment so that the question “which bytes are running” has an answer that does not depend on a registry’s current state.