Registry Protocols: Blob Uploads, Ranges and Cross-Repository Mounts

A registry is an HTTP server with a small number of endpoints and a large number of ways to get them subtly wrong. The protocol is documented as an API rather than a state machine, so the parts that matter most, resumability, redirects, and the ordering of validation, are learned from the specification’s edge cases and from how clients behave when a server deviates.

The endpoint surface

Four path prefixes carry almost everything. /v2/ is a version check that returns 200 or 401 and, in the latter case, a WWW-Authenticate header naming the token service. /v2/<name>/blobs/<digest> fetches or checks a blob. /v2/<name>/blobs/uploads/ starts an upload. /v2/<name>/manifests/<reference> handles manifests, where the reference is a tag or a digest.

<name> is the repository path and may contain slashes, so the routing has to distinguish /v2/team/service/blobs/... from /v2/team/blobs/... by parsing from the right. This is a frequent source of 404s against registries that assume a single path segment.

A HEAD on a blob path is the existence check, and it is the most-executed request in the protocol. A client pulling an image asks HEAD for every layer before deciding to download, so the cost of a HEAD bounds pull latency for already-cached images, and the answer is keyed by the layer digest discussed in Content Addressing: Why a Layer Is Named by Its Digest. The response carries Content-Length and Docker-Content-Digest, and a well-behaved client verifies that the digest in the header matches the one it asked for. A registry that returns 200 for a digest it does not have, or that omits the digest header, breaks caching assumptions in ways that only appear when a layer is corrupt.

Uploads are sessions, not PUTs

A blob upload is a three-step dance because blobs are large and networks are unreliable. POST /v2/<name>/blobs/uploads/ creates an upload session and returns 202 with a Location header containing a session identifier. The client then either PATCHes chunks to that location or PUTs the whole blob to it with the digest as a query parameter. A final PUT with ?digest=<digest> commits; the server verifies the received bytes against the digest and only then makes the blob visible.

The session is server-side state, and its lifetime is a policy, not a specification constant. Registries expire sessions after an idle period, and a client that pauses too long receives 404 on the next PATCH and must restart. Nothing in the protocol negotiates that timeout, so a client must be prepared to restart an upload at any point.

Chunked uploads use Content-Range on PATCH. The range is inclusive on both ends, and the server responds with Range: 0-<offset> in the Location or a Range header, naming the last byte it has durably accepted. The subtlety is that the server may accept fewer bytes than sent: a PATCH of 10 MB can be acknowledged as having advanced the offset by less, and the client must resume from the acknowledged offset rather than from what it wrote. Clients that assume their own byte count is the server’s lose data and produce a digest mismatch at commit time.

import urllib.request

# Resume-aware upload loop; the server's Range header is authoritative.
offset = 0
with open(blob_path, "rb") as f:
    while True:
        f.seek(offset)
        chunk = f.read(4 * 1024 * 1024)
        if not chunk:
            break
        end = offset + len(chunk) - 1
        req = urllib.request.Request(
            upload_url, data=chunk, method="PATCH",
            headers={"Content-Type": "application/octet-stream",
                     "Content-Range": f"{offset}-{end}",
                     "Authorization": f"Bearer {token}"})
        with urllib.request.urlopen(req) as r:
            rng = r.headers.get("Range")
        # Trust the acknowledged offset, not len(chunk).
        offset = int(rng.rsplit("-", 1)[1]) + 1 if rng else end + 1

A monolithic PUT is simpler and works for small blobs, but it forfeits resumability: an interrupted transfer restarts from zero. For a 500 MB layer over a link with intermittent failures, the difference is the difference between finishing and not.

Redirects and the storage backend

Large registries do not proxy blob data through the API server. They return 307 with a Location pointing at object storage, and the client fetches from there directly. The redirect is deliberate: it keeps the API server’s memory and bandwidth out of the data path, and it lets the storage service handle range requests, caching, and its own authentication.

Two things break under redirects. The first is authentication: the token for the registry is not valid for the object store, so the signed URL carries its own credentials and expires quickly. A client that caches the URL and reuses it later gets 403. The second is that HEAD requests may be redirected too, so a client’s existence check goes to the object store and inherits its consistency behaviour. Object stores are commonly eventually consistent for listings but strongly consistent for reads by key, which is why digest-addressed storage works on them at all: the key is the content, so there is no “latest” to be stale.

The Location header on upload sessions has the same property. It may be absolute, relative, or a URL on a different host, and clients must resolve it rather than assume the registry’s own host. A registry behind a reverse proxy that rewrites Location incorrectly produces clients that upload to the wrong place, which manifests as a 404 on the session.

Manifests, validation and the ordering of checks

A manifest PUT is validated before it is accepted. The server checks that the media type is one it supports, that the JSON parses, that every referenced blob exists in the repository, and that the manifest’s schema is coherent. The existence check is the expensive part: a manifest with a hundred layers requires a hundred lookups, and a registry that skips the check accepts a manifest referencing blobs it will never be able to serve.

This is why push order matters. Layers must be uploaded and committed before the manifest that references them, and the manifest’s descriptor list is the structure defined in From Image Manifest to Mounted Root Filesystem. A client that pushes the manifest first gets a 400 or a 404 naming a missing blob, and the error is not a retryable failure: the client must upload the missing layer and push again.

Validation is also where tag immutability is enforced, if the registry implements it. A policy that forbids overwriting a tag makes the second PUT to that tag fail with a conflict, and the client must choose a new tag. The protocol itself has no notion of immutability; it is a server policy expressed through status codes.

A manifest that references a manifest, rather than a layer, is an index or manifest list. It carries a mediaType naming the index type and an array of platform-tagged descriptors. Pushing an index requires the referenced manifests to exist in the same repository, so a multi-platform push is a fan-out: build each platform’s manifest, push each, then push the index that names them all. The index’s digest is what a tag resolves to, and a client selects the entry matching its platform from the array. A client that cannot find a matching entry fails rather than guessing, which is why an index missing an architecture produces “no matching manifest” on that platform.

Cross-repository mounts

When a pushed layer already exists in another repository on the same registry, transferring it again is waste. The mount shortcut is a POST to the target repository’s upload endpoint with a mount query parameter naming the digest and a from parameter naming the source repository. If the registry has the blob and the client is authorised to read the source, it links the blob into the target repository and returns 201 with the blob’s location, without any data transfer. If either condition fails, it returns 202 with an upload session, and the client falls back to uploading.

The authorisation check is the whole point. A client that can mount any digest from any repository can read any blob the registry holds, which would defeat repository-level access control. The check must be against the client’s pull permission on the source, and a registry that only checks the target’s push permission is a confused-deputy vulnerability.

Deduplication then happens at the storage layer. The blob exists once, and two repositories hold references to it. Deleting from one repository removes the reference, not the blob; the blob is removed when no repository references it, which is why garbage collection has to consider repository membership rather than global reachability alone.

Consistency and the deleted-but-cached blob

A registry must not serve a blob after it has been deleted, and clients cache aggressively. HTTP caching rules apply: HEAD and GET responses for digest-addressed content can carry long max-age values, because the content at a digest never changes. A deleted blob is then a cache entry that is stale but still marked fresh, and a client can serve it from its own store even though the registry no longer has it. The Docker-Content-Digest header does not help; it confirms the content, not its continued existence.

The reverse problem is a tag that moves. Tag responses are typically no-cache, so a client revalidates and receives the new digest. A registry that caches tag responses for any duration serves stale tags, and clients pull an image that is no longer current. The two namespaces therefore want opposite caching policies, and a server that applies one policy to both gets one of the two wrong.

Failure modes worth naming

An upload session whose blob is committed but whose manifest push fails leaves an orphaned blob, reachable only by digest. Repeated builds accumulate these, which is why registries expose a garbage collection pass and why clients that push by digest benefit from a retention policy.

A PATCH that is retried without checking the acknowledged offset duplicates bytes, and the commit fails on digest verification with no indication of where the duplication occurred. The digest is the only error signal, and it arrives at the end.

A HEAD that returns 200 with a Content-Length different from the descriptor’s size indicates a registry that is serving something other than what was asked for; a client that skips the size check proceeds to download the wrong number of bytes and fails at digest verification instead of immediately.

And a 307 to a URL whose host resolves differently for the client than for the registry produces an upload that never lands, with the session expiring and the client retrying forever. Redirect targets are part of the registry’s configuration surface, and clients cannot correct them.