Pull-Through Caches and Where the Trust Anchor Sits

A pull-through cache sits between a cluster and the registries it pulls from, and its stated purpose is bandwidth and availability: fetch each blob once, serve it many times, keep working when the upstream registry is slow or unreachable. Those are real benefits and they are the reason the component exists. The security question it raises is different, and it is usually answered by accident rather than by design. A cache is a party that serves the bytes a verifier checks, and a verifier that trusts content because it came from the cache has moved its trust anchor without saying so.

What a proxy changes about the set of parties

Without a cache, a pull involves the registry, its token service, and whatever object storage it redirects to. With a cache, the client’s requests go to the cache, and the cache’s requests go upstream. The client no longer sees the upstream registry’s authentication challenge, its redirect behaviour, or its rate limits, and it no longer knows whether the response it received was served from cache or proxied.

The client’s digest check still applies, and that is the property that makes caching compatible with content addressing. A cache cannot substitute content for a digest without failing the client’s hash. It can, however, choose which digests it serves, and it can choose what a tag resolves to. A cache that resolves library/nginx:1.27 on the client’s behalf is performing the one step in the pull path that is not hash-verified, and it is performing it out of the client’s sight.

This is the crux. Digest verification protects the content of every object named by a digest. Tag resolution has no such protection. A cache that resolves tags is as trusted as the registry it proxies for, and a cache that has been configured to resolve tags from a different upstream than the client believes has silently replaced the registry.

The two-level key and why negative caching is dangerous

Cache entries are keyed by repository name and digest for blobs, and by repository name and tag for tag resolution. The blob key is safe because the digest is in the key: two requests for the same digest must return the same bytes, and a mismatch is detectable.

Tag resolution is a different kind of entry. It is a mapping from a name to a digest, it has no verification, and its lifetime is a policy decision. Caches store it with a TTL, and the TTL determines how long a repointed tag continues to resolve to the old digest. During that window, a client that asks the cache for a tag receives a stale answer, and the pull that follows is internally consistent (the digest it resolves to is real, the content matches) but it is not what the upstream registry would have said.

Staleness in the safe direction is an availability feature: the upstream is down, the cache serves a known-good digest, deployments continue. Staleness in the unsafe direction is a supply-chain problem: a tag was repointed to a patched image in response to a vulnerability, and the cache keeps serving the old digest. The window is bounded by the TTL, and the correct value depends on how quickly the organisation needs to propagate tag movements, which is a different number from how long it wants to survive an outage.

Negative caching is worse behaved. A cache that records “this digest does not exist” or “this repository has no referrers” for a TTL will continue to report absence after the artifact is published. In a verification pipeline, absence is a denial: the signature exists upstream, the verifier asks the cache, the cache answers that there are no referrers, and admission rejects the workload. The failure looks like a missing signature rather than a stale cache entry, which makes it expensive to diagnose. Short negative TTLs, or no negative caching for referrer queries, avoid it.

The trust anchor must not travel through the cache

A verifier needs three things: the artifact, the signature or attestation about it, and the public keys or certificate roots it will accept. The first two legitimately come through the cache, because both are content addressed and both are checked against digests the verifier already knows or obtains from the signature.

The third must not. If the key set is fetched from the same registry that serves the image, then a party controlling that registry controls both the claim and the standard by which the claim is judged. They can publish a signature over a malicious image and publish the key that verifies it, and a verifier that fetches its trust root from the registry will accept both. This is not a hypothetical weakness in a protocol; it is the definition of what a trust anchor is supposed to prevent. The anchor has to arrive through a channel the artifact’s publisher cannot write to.

In practice the anchor is one of: a key set embedded in the verifier’s configuration and updated through the organisation’s normal configuration management; a certificate root that is part of the operating system’s trust store; or a policy bundle distributed and signed by a separate system with its own root. The first is the most common and the weakest operationally, because a static key set has no expiry and no revocation channel; rotating a compromised key means editing configuration on every verifier.

Delegation and the offline verification problem

Verification at pull time wants to be offline. A node pulling an image should not need to reach a key server, both because the key server becomes an availability dependency and because the request reveals which images are being deployed. The standard structure for offline verification is delegation: a long-lived root key signs a short-lived delegation to a signing key, and the verifier holds the root and checks the delegation chain.

The delegation carries an expiry. A verifier that has been offline since before the expiry cannot distinguish a delegation that expired from one that was revoked, and the conservative behaviour (reject) turns an offline node into a node that cannot pull. The permissive behaviour (accept until the next successful refresh) extends the validity of a possibly-revoked key for the duration of the outage. Both are defensible, and the choice has to be made explicitly, because the default in most implementations is one of them without documentation.

RFC 5280’s treatment of certificate validity and revocation is the reference point for how this is normally reasoned about: a certificate has a not-before and a not-after, revocation is published separately, and a verifier that cannot obtain revocation information has to decide whether to proceed. The image-signing case inherits the same structure and the same unresolved question, with the added complication that the artifact being verified is usually more valuable than the certificate.

The cache is an observer

A pull-through cache sees every image the cluster pulls, every repository name, and the timing of every request. That is a complete inventory of what the cluster runs and when it changes. Whether this is acceptable is an organisational question, but it is worth stating because it is often not considered: a cache deployed for bandwidth reasons becomes a monitoring point with better visibility into workload composition than most monitoring systems have.

The cache also becomes a single point of failure for both availability and integrity. If it is compromised, it can serve any content for any digest it chooses to lie about (which fails verification) and it can serve stale tag resolutions, which do not fail verification. The asymmetry means an attacker who controls a cache gets the most value from manipulating tag resolution and referrer listings rather than blob content, because those are the paths without hash checks.

Designing the placement

Three arrangements are common, and they differ in where the trust decision happens.

Cache as a transparent mirror, clients verify. The cache resolves tags and serves blobs; clients pin digests and verify signatures themselves. The cache’s influence is limited to tag resolution, which clients avoid by pinning, and to availability. This is the arrangement with the smallest trust surface, and it requires that clients be configured to pin, which is a policy that has to be enforced elsewhere.

Cache as a policy point. The cache verifies signatures before serving, and refuses to serve content that fails. This centralizes policy and removes the need to configure every node, at the cost of making the cache part of the trust boundary: a compromised cache now decides what is acceptable, and its decisions are invisible to clients that trust its output. A cache in this role must sign or otherwise attest to what it served, or clients must re-verify.

Cache as an origin for air-gapped environments. The cache is populated from upstream by an explicit sync process, and the cluster pulls only from the cache. Here the cache is the registry, with all the authority that implies. The sync process is the supply-chain control, and its records (what was synced, from where, at which digest) are the provenance for everything the cluster runs. Air-gapped environments often have the strongest verification requirements precisely because they cannot re-check against upstream later.

In all three, the question to answer explicitly is which component is allowed to make an unverified assertion. Tag resolution is always such an assertion. Referrer listings are such an assertion. Key distribution must never be. Separating those three by where they are served is the design work, and it is more consequential than the caching configuration itself.

The verification checks that a client applies to what the cache returns are the same ones it would apply to a registry, and they are set out in What an Image Signature Covers and What It Leaves Out and Admission-Time Verification and Digest Pinning. What changes with a cache is not the checks but the identity of the party they are applied against, and whether the material used to perform them arrived through a channel that party controls.

The last remaining question in this chain is whether the artifact can be regenerated independently, which is what turns a set of attestations into evidence. Verifying Provenance by Rebuilding covers that comparison and the packaging details that determine whether it succeeds.