Registry Index Metadata and Resolution Without Downloading Artifacts

Resolution needs three facts about every candidate version: that it exists, what it depends on, and what bytes it corresponds to. A registry can supply all three at several granularities, and the difference between granularities is the difference between a resolver that answers in one round trip per package and one that downloads hundreds of megabytes to read a few kilobytes of dependency lists.

Three shapes of index

The oldest shape is a per-package page listing files. PEP 503 defines an HTML page per normalised project name, containing anchors whose text is a filename and whose href is the file location. Everything else must be inferred from filenames: the version, the artifact type, the platform tags. Beyond the interpreter-range hint that PEP 503 requires on each anchor, dependency information is absent, so a resolver must download a distribution and read its metadata to learn what it requires. PEP 691 added a JSON representation of the same page, selected by content negotiation, so a client can parse the same information without an HTML parser; the underlying resource and its cacheability are unchanged, only the encoding.

The second shape is a per-package document containing metadata for every version. npm’s registry returns a packument: a JSON object with a version map, where each entry carries its dependency ranges, peer ranges, optional ranges, and the tarball URL plus integrity hash. One request yields everything needed to resolve, and the abbreviated form of the packument, selected by an Accept header, omits the readme, maintainer history, and other fields the resolver does not need, which makes the response substantially smaller on packages that have accumulated a long publication history.

The third shape is a per-package file of records, one per line, appended as versions are published. The Cargo registry index uses this layout: the path to a package’s index file is derived from its name by a fixed prefix rule (one, two, or three characters, then the name), and each line is a JSON object giving the version, its dependency list with requirement strings, a checksum, feature definitions, and a yanked flag. Because a publish only appends a line, a client that has already read a file needs just the new suffix when a version is added. The yanked flag is the exception: a registry may change it at any time, which rewrites that record in place, so a client that cares about yank state cannot treat the file as strictly append-only.

Each shape is a trade. The HTML page is trivial to serve from static files and trivial to cache, but pushes metadata into the artifact. The packument is efficient per request but makes the response grow without bound for popular packages, which is why npm’s abbreviated variant exists. The append-only log is the most efficient for incremental updates and the least convenient for a client that wants a single consistent snapshot, since reading the file while a publish appends can yield a partial view unless the client handles the trailing newline carefully.

Metadata-only resolution and the sidecar file

The fundamental inefficiency of the HTML shape is that dependency information lives inside an archive. A wheel is a ZIP file with a metadata directory containing a file that lists Requires-Dist entries; an sdist is a compressed tarball with the same information in a different place, sometimes only computable by running a build. Downloading a multi-megabyte archive to read a few kilobytes of dependency names wastes bandwidth and, worse, turns resolution into a sequence of large transfers that cannot be parallelised cheaply or cached by hash.

PEP 658 addresses this for wheels by allowing the index to serve the metadata file alongside the distribution. The index page gains a data-dist-info-metadata attribute on the anchor, whose value is either a hash of the metadata file or the bare string true, and the client can fetch <file-url>.metadata directly. The hash form lets the client verify that the metadata it received is the metadata for that exact wheel, which prevents a stale or substituted sidecar from influencing resolution; the boolean form is for indexes that cannot supply a hash. The attribute has since been renamed data-core-metadata to reflect that it covers the core metadata file rather than a particular directory name, and clients accept both spellings. Clients that understand neither simply download the wheel, so the extension is backwards compatible.

The effect on a resolver is structural. With sidecar metadata, the resolve phase touches only small text resources; the download phase touches archives, and only for the versions actually selected. A resolver can therefore evaluate hundreds of candidate versions with a few hundred kilobytes of transfer, which changes the cost model of a wide search like the one described in Dependency Resolution as Backtracking Search Over Version Sets.

Source distributions remain the hard case. Their dependency metadata may be declared in a static file, in which case it can be served the same way, or it may be generated by the build backend at build time, in which case the only way to learn it is to run the build. Resolvers handle this by preferring wheels for metadata purposes even when a source build will ultimately be used, and by treating a version whose metadata cannot be obtained without a build as a last resort.

Caching is part of the protocol

An index that is fetched on every install is a single point of failure and a latency floor. Every serious client caches, and the caching semantics are specified by HTTP rather than by the package manager.

Index documents are cacheable but mutable: a new version can be published at any time, so a client must revalidate. Conditional requests with If-None-Match against the ETag the server supplied, or If-Modified-Since against Last-Modified, let the client pay for a round trip but not for the body when nothing changed. RFC 9111 defines the relevant rules, including that a response to a conditional request that returns 304 carries no body and refreshes the stored entry’s freshness.

Artifact files are the opposite: a published version’s bytes are supposed to be immutable, so an artifact URL can be cached for a long time, ideally forever. Registries enforce this by rejecting re-uploads of an existing version, and clients rely on it by treating a hash match as sufficient proof of correctness. Where immutability is not enforced, the hash in the lockfile becomes the only defence, which is why lockfiles carry hashes at all, as covered in Lockfiles as Reproducibility Contracts.

The split between mutable index and immutable artifact also determines what a mirror must do. A mirror that caches index pages for minutes will delay visibility of new releases; a mirror that caches them for days will appear broken to a publisher. A mirror that caches artifacts indefinitely is safe as long as it keys by URL and verifies hashes.

Cargo’s sparse index adds a refinement worth noting. Each package’s index file is served over HTTP on its own URL, so a client fetches only the files for packages that appear in its graph, and revalidates them with conditional requests on subsequent runs. That is a different caching shape from a single repository-wide index fetched as one object: the per-package granularity means a resolution touching forty packages costs forty small conditional requests rather than one large transfer, and a package the graph does not mention costs nothing.

Failure modes that the metadata layer creates

Four are common.

Truncated or interleaved responses. A client reading an append-only log can observe a partially written last line. Parsers must tolerate a trailing incomplete record and must not treat it as corruption; the correct behaviour is to stop before it and refetch later.

Version lists that disagree with artifacts. An index may advertise a file that has since been removed, or an artifact may exist that the index does not list. Clients that trust only the index will fail with a clear error on the first case; clients that probe URLs directly will produce confusing partial installs. Listing is authoritative; probing is not.

Metadata that disagrees with the artifact. A sidecar file whose hash is not checked can describe a different dependency set than the archive contains. Any client that consumes sidecar metadata must verify the advertised hash, or it has replaced a trustworthy but slow path with a fast but forgeable one.

Rate limiting expressed as success. Some registries return a 200 with an error body, or a redirect to a login page, when a client exceeds a quota. A parser that assumes JSON on every 200 will crash on HTML. Content-type checking is not optional in a client that talks to many registries.

A minimal client for the append-only shape shows how small the protocol surface is:

import json, urllib.request

def index_path(name):
    n = name.lower()
    if len(n) == 1: return f"1/{n}"
    if len(n) == 2: return f"2/{n}"
    if len(n) == 3: return f"3/{n[0]}/{n}"
    return f"{n[:2]}/{n[2:4]}/{n}"

def versions(base, name):
    url = f"{base}/{index_path(name)}"
    with urllib.request.urlopen(url) as r:
        body = r.read().decode("utf-8")
    out = []
    for line in body.splitlines():
        try:
            rec = json.loads(line)      # skip a partially written tail
        except json.JSONDecodeError:
            break
        out.append((rec["vers"], rec.get("yanked", False), rec.get("deps", [])))
    return out

The function ignores caching, checksums, and features, but it captures the property that makes the format work: a version list is a sequence of small records, and the resolver can read all of them without touching an archive.

What the index cannot tell you

An index describes versions, not resolutions. It does not know which combination of versions is installable on the current platform, because that depends on artifact tags rather than on metadata. It does not know whether a version’s build will succeed, because that depends on a toolchain. And it does not know whether a version is safe, which is a question about provenance rather than about dependency ranges.

Those three gaps are why resolution has a second phase that consults platform compatibility, described in Platform Tags and Binary Compatibility in Distributed Artifacts, and why publication-time controls exist separately from resolution-time metadata.