The Fetch Protocol: Wants, Haves, and Thin Packs

A fetch has to move the objects a client lacks and nothing else. The client knows what it wants by name, and it knows some of what it already has, but it does not know the server’s history, and the server does not know the client’s. The protocol is the conversation that closes that gap: the server advertises what it has, the client declares what it wants, both sides exchange commit identifiers until they find common ground, and the server sends a pack containing everything reachable from the wants but not from the common commits. Every message in that conversation is framed the same way, and the framing is the part most worth understanding first.

Frames, not streams

Messages are carried as pkt-lines. A pkt-line starts with four hexadecimal digits giving the total length of the packet including those four bytes, followed by that many bytes minus four of payload. The maximum length is 65520 bytes, so a payload can be at most 65516 bytes; anything longer has to be split by the sender.

Three lengths are special. 0000 is a flush packet, which terminates a section of the conversation. 0001 is a delimiter, used in the newer protocol to separate parts of a request. 0002 is a response-end marker. All three consist only of their four-digit header. Any other value below four is invalid.

Framing at this level means the transport can be a pipe, a socket, or a series of HTTP request bodies without changing the message grammar. It also means the receiving side must handle a message split across TCP segments and two messages arriving in one read, which is why implementations read four bytes, parse the length, then read exactly that many more.

def pkt_line(payload: bytes) -> bytes:
    if len(payload) > 65516:
        raise ValueError("payload exceeds pkt-line maximum")
    return b"%04x" % (len(payload) + 4) + payload

FLUSH = b"0000"
DELIM = b"0001"

The advertisement

In the original protocol the server opens with one line per reference, each of the form <identifier> <name> followed by a NUL and a list of capability names on the first line only. Capabilities describe what the rest of the conversation may use: side-band multiplexing, which interleaves progress messages with pack data on separate channels; thin packs; offset deltas; shallow history; the ability to request a filtered pack; and the agent string. A symref capability reports what symbolic references point at, so the client learns the remote’s default branch without a second round trip.

The client then sends its wants, one per line, naming identifiers it does not have. Wants must be advertised unless the server is configured to allow arbitrary identifiers, which is why fetching an unadvertised commit from a server that has pruned the ref fails even though the object is still present. After the wants comes a flush, and then negotiation begins.

The newer protocol restructures this into commands. The server advertises a version and a capability list, and the client issues requests such as a reference listing with prefix filters, or a fetch with explicit wants, haves, and options. Each request is a separate exchange, which makes the protocol stateless in the HTTP sense: every request carries everything needed to answer it, so a proxy or a load balancer can route consecutive requests to different backends.

Finding common history

The purpose of negotiation is to bound what the server must send. Reachability is the criterion: the server excludes everything reachable from any commit the client is known to have. Sending a commit identifier is therefore a claim about a whole subgraph, and one well-chosen identifier can eliminate most of the work.

The client sends have lines naming commits it possesses, typically the tips of its branches and then older commits in an order derived from its own history. The server replies with acknowledgements. In the simplest mode it acknowledges the first common commit it finds and the client stops. In the multi-ack mode it reports each common commit it discovers, and in the detailed variant it distinguishes three states: a commit that is common but not yet sufficient to stop, a commit that is common and sufficient, and a final acknowledgement after the client declares it is done.

Batching matters because each round trip costs latency. The client sends haves in groups and the server answers the group before the next one is sent. The number of rounds is bounded by the divergence between the two histories: a client that is one commit behind needs one round, while a client whose history shares nothing with the server may exhaust its candidates before finding common ground, at which point the server sends everything.

Ordering the haves well is an optimisation problem in itself. Walking the client’s history newest-first finds a common commit quickly when the client is close to the server, but a client with a long diverged branch may spend many rounds offering commits the server has never seen. Alternative strategies sample commits at increasing distances to bracket the divergence faster, trading precision for fewer rounds. The choice is a client-side configuration, and the wrong choice shows up as a fetch that transfers the right data after an unreasonable number of round trips.

Thin packs and the repair step

The pack the server sends is built from the objects it decided to include, and delta compression inside that pack can use objects the client already has as bases. Doing so saves the server from sending a full copy of a large file when the client holds a near-identical version. The result is a thin pack: a pack whose deltas reference bases that are not in the pack.

A thin pack is not a valid pack on its own. The client repairs it by looking up each missing base in its own object store, appending the base to the pack, and recomputing the trailing checksum and the index. If a base is missing locally, the repair fails and the fetch must be retried without thin-pack support. This is the mechanism by which a server can prune an object and break a client that was relying on having it.

The same interaction explains why a fetch that fails partway can leave a temporary pack in the object directory. The pack is written before it is indexed and before the references are updated, so an interrupted fetch leaves unreferenced data that the next collection treats as garbage.

Filters, shallow boundaries, and missing objects

A filtered fetch asks the server to omit classes of objects. A blob filter sends commits and trees but no file contents; a tree filter sends commits only. The resulting repository is a partial clone: the graph is complete enough to check out, but reading a file requires fetching its blob from the promisor remote on demand.

A shallow fetch takes a different approach, cutting history at a specified depth and recording the boundary commits in a file. Commits beyond the boundary do not exist locally, so traversals must stop there, and operations that need full ancestry refuse to run. Both mechanisms reduce transfer at the cost of breaking the invariant that an object’s transitive closure is present, which is why so much of the rest of the system has to know about them.

The wire format also carries the side-band channels. When side-band is negotiated, the pack data arrives on one channel and progress or error text on another, each message wrapped in a pkt-line with a one-byte channel number. This is what allows a clone to display progress without corrupting the pack stream, and what allows an error to be reported in the middle of a transfer.

What breaks

The failure modes are mostly mismatches between what one side assumes and what the other does. A server that has pruned an object a client wants fails the want with an error naming the object rather than sending an incomplete pack. A client that offers haves the server cannot parse, because it is speaking a different protocol version, gets a NAK for every batch and eventually transfers the full history. A proxy that buffers responses breaks the progress channel’s timing without breaking correctness, while one that rewrites chunked encoding can break framing entirely.

Negotiation also interacts with time. Commit dates order the search, and a client with a badly skewed clock offers commits in an order that misleads the server’s heuristics; the result is correct but slow. None of this is visible in the object graph, which is the point: the protocol’s job is to transfer a set of content-addressed objects that the receiver can verify, and every optimisation above is about reducing the size of that set without changing what it means.

What arrives is a pack, and what the client does with it afterwards is a matter of indexing it and updating references atomically, so that a reader never sees a branch pointing at an object that has not finished arriving.