Content Addressing and What Artifact Hashes Actually Prove

A package manager that verifies a SHA-256 digest before unpacking an artifact has done something real and something much smaller than it appears. It has confirmed that the bytes it received are the bytes some other party expected. It has not confirmed who that party was, whether they intended those bytes, or whether the bytes are safe. The gap between those statements is where most supply-chain reasoning goes wrong.

Digest, identifier, and signature are three different claims

A digest is a function of content. Given the same bytes, SHA-256 produces the same 32-byte output; given different bytes, it produces a different output with overwhelming probability. That property makes a digest a name for a byte sequence, which is the basis of content addressing: a store keyed by digest can be written once and read by anyone who knows the key, and a reader can verify that what it got is what the key designates.

An identifier is a digest used as a lookup key in a trusted index. When a lockfile records "integrity": "sha512-..." for a tarball, the hash is an identifier whose binding to a package name and version was established when the lockfile was generated. The security of that binding rests entirely on the trustworthiness of the index the client fetched from at generation time. If the index can be induced to serve a hash for attacker-supplied bytes, verification passes and the attack succeeds, which is why mirroring and index authentication matter as much as artifact hashing.

A signature is a claim by a key holder. An Ed25519 or RSA signature over an artifact says that whoever controls a private key asserts something about those bytes. Verification requires a public key and a policy about which keys are acceptable for which packages. Signatures survive a compromised index; digests do not.

Registries generally provide the first two and are increasingly providing the third, but the three are often conflated in tooling: a client that says it “verifies integrity” may be checking a hash it fetched from the same server that served the artifact, which is a transport check rather than an authenticity check.

Why verification still catches real problems

The narrow claim is still worth a great deal. Digest verification catches:

Corrupted transfers. A truncated or bit-flipped download fails the digest, and the client retries rather than unpacking a broken archive. Without the check, the failure surfaces later as a confusing build error inside an unrelated file.

Mirror substitution. A mirror that serves different bytes than the origin is caught if the expected digest came from the lockfile rather than from the mirror. This is the main reason lockfiles carry hashes per artifact rather than per package.

Concurrent republication. A registry that allows replacing an artifact under the same version creates a window where two clients get different bytes for the same coordinate. A digest in the lockfile converts that from silent divergence into a hard error on one side.

Those are transport- and consistency-level guarantees. They do not address a malicious publisher, a compromised maintainer account, or a registry that is itself the attacker. For those, the mechanism has to be a signature with a key the client trusts independently, or a transparency log that makes misbehaviour detectable after the fact. Certificate Transparency, defined in RFC 6962, established the pattern for certificates: append-only logs with cryptographic proofs of inclusion, so a log operator cannot quietly show different entries to different observers. The Update Framework takes the adjacent problem, where an attacker replays an old but validly signed metadata set or withholds a newer one, and answers it with role separation, version numbers, and expiry. Neither mechanism is about the hash; both are about the metadata that says which hash is current.

The same digest syntax appears outside package managers. Subresource Integrity, a W3C specification, defines an integrity attribute whose value is a space-separated list of algorithm-base64 tokens, with the same multi-algorithm structure and the same verification rule: the browser accepts the resource if any listed digest matches. The convention is worth recognising because it makes digest strings portable between tools, and because it shows the same weakness in a different setting, where the expected digest is embedded in a page that is itself fetched over the network.

Choosing the digest and the encoding

Algorithm choice is mostly settled. MD5 and SHA-1 are unsuitable for any adversarial setting: MD5 collisions are cheap to construct, and SHA-1 chosen-prefix collisions are practical. SHA-256 and SHA-512 are the defaults; SHA-512 is generally faster on 64-bit hardware because its compression function operates on 64-bit words, while SHA-256 has a shorter encoding and is accelerated by dedicated instructions on CPUs that provide them.

Encoding is a separate decision, and it is where interoperability breaks. A digest can be written as lowercase hex, uppercase hex, base64, or base64url. npm’s integrity strings use base64 with an algorithm prefix and optional multiple algorithms separated by spaces, so a single field can carry both SHA-512 and SHA-256 and let the client pick the strongest it supports. Python’s --hash requirements use hex with the algorithm prefix, as in sha256:.... Tools that consume both must not assume one.

The multi-algorithm form exists for migration: a publisher can add a stronger digest while old clients still verify the weaker one. It also creates an ambiguity that implementations must resolve explicitly, since “verify all listed digests” and “verify the strongest supported digest” have different failure behaviour when one algorithm’s digest is wrong.

A canonical implementation is short:

import hashlib, base64

STRENGTH = {"sha512": 3, "sha384": 2, "sha256": 1}

def check(data, spec):
    # spec like "sha512-abc... sha256-def..." (npm style, base64)
    items = []
    for token in spec.split():
        algo, _, encoded = token.partition("-")
        if algo in STRENGTH:
            items.append((STRENGTH[algo], algo, base64.b64decode(encoded)))
    if not items:
        return False
    _, algo, expected = max(items)        # verify the strongest one present
    return hashlib.new(algo, data).digest() == expected

Two details matter. hashlib.new requires the algorithm name in the form the runtime knows, so a spec that says sha512 works while one that says SHA-512 may not. Choosing the strongest listed algorithm rather than checking every one is what makes the multi-algorithm form a migration path: a stale weaker digest left in a manifest does not cause a false failure, while a tampered artifact still fails against the strongest digest the client understands. And comparing digests with != on bytes is not constant-time, which is irrelevant for a public artifact digest and relevant for anything secret-derived; comparison timing is a real concern for MACs, not for content hashes.

The unpacking problem

Verification happens on the artifact as transferred, not on what the artifact contains. That gap is exploited routinely, because archive formats have structure beyond a flat byte sequence.

A tar archive is a sequence of 512-byte headers and padded file bodies. A gzip stream wrapping it is a sequence of deflate blocks with a trailer. The digest covers the compressed bytes exactly, so any transformation of the compression, such as recompressing at a different level, changes the digest while producing identical extracted content. This means the digest identifies the transfer, not the logical package, and two registries that store the same package with different compression produce different digests for identical contents.

Inside the archive, path traversal is the classic defect. An entry named ../../etc/cron.d/job extracts outside the destination directory unless the unpacker normalises and rejects such names. An entry that is a symlink pointing outside the tree is a variant of the same attack. Neither changes the archive digest, because the malicious name is part of the legitimate bytes the publisher signed or the index listed.

Absolute paths, hard links to files outside the tree, device nodes, and entries with the setuid bit are further cases. A careful unpacker rejects all of them, or extracts with restrictive permissions and refuses links that escape the root. Python’s tarfile module gained a data filter for exactly this class of problem, and extraction modes that default to it are safer than the historical permissive behaviour. The lesson generalises: hash verification and safe extraction are independent defences, and a system with only the first is not protected against a malicious artifact that was legitimately published.

Content addressing as storage design

Once artifacts are keyed by digest, storage becomes simpler. A cache directory whose entries are named by digest can be shared between projects without collisions, because two different packages cannot have the same key unless they have the same bytes. Deduplication falls out for free: two projects depending on the same version of the same package store one copy.

The design has two costs. The first is that a digest is not human-meaningful, so a cache needs a separate mapping from package coordinates to digest, which is exactly what the lockfile provides. The second is that digest-keyed storage is immutable by construction: an entry cannot be updated, only added or removed. That is a feature for artifacts and an obstacle for anything that legitimately changes, which is why index metadata is cached by URL and validator rather than by content hash, as described in Registry Index Metadata and Resolution Without Downloading Artifacts. The mapping from coordinates to digest is what Lockfiles as Reproducibility Contracts records, and a lockfile without per-artifact hashes has given up precisely this property.

Build systems that content-address outputs, rather than only inputs, extend the idea further: an action’s output is keyed by a hash of its inputs, its command line, and its environment, so a rebuild with identical inputs can be skipped. That works only when the action is deterministic, which archive creation is not by default because timestamps and file ordering vary. Reproducible builds address this by fixing those fields, which is what makes two independent builds of the same source comparable at the digest level at all. That comparability is the foundation for the verification approach described in Reproducible Builds and the Sources of Byte-Level Nondeterminism.

The honest summary: a digest proves the bytes are the bytes someone expected. Everything stronger requires knowing who expected them, and under what policy they were allowed to.