Model Artifact Identity: Why a Checkpoint Hash Is Not a Deployment

A checkpoint file is a mapping from parameter names to tensors. The behaviour a request observes is a function of considerably more than that mapping. The tokenizer that turns bytes into ids, the chat template that wraps those ids into a prompt, the preprocessing code that normalises an image or fills a missing feature, the numeric format the weights were converted to, the maximum sequence length the server enforces, and the sampling defaults all sit between the request and the response. Any one of them can change without a single tensor changing, and several of them change behaviour as sharply as retraining would.

The operational consequence is that “which model is deployed” is not answerable by a file hash. It is answerable only by a tuple, and the tuple has to be recorded, addressed, and compared as a single object. Systems that treat the checkpoint as the identity of a deployment spend their incident reviews reconstructing the rest of the tuple from logs.

The deployed unit is a tuple

Consider what has to agree for two servers to return the same output for the same input at temperature zero. The weights must match bit for bit or close enough that rounding does not flip an argmax. The tokenizer must produce the same id sequence, which means the vocabulary file, the merge rules or byte-level encoder, and the special-token table must all match. The prompt assembly must match, and in instruction-tuned models that assembly is a template string with role markers and generation-prompt suffixes stored beside the tokenizer. The preprocessing for non-text inputs must match, including resize interpolation, channel order, and normalisation constants. The runtime must apply the same truncation policy and the same dtype for the compute path, because a model converted to a narrower format produces different logits.

Four of those six are code or configuration, not learned parameters. In a typical repository layout the weights are the largest object and the only one under a formal versioning scheme, while the template string lives in a file that is edited in place and the normalisation constants live in a module that is imported by both the training job and the server.

The tuple is the deployment identity. Everything downstream, including the registry record, the rollout plan, and the rollback target, refers to the tuple and not to the checkpoint.

Content addressing stops at the boundary drawn around it

A content-addressed store names an object by the digest of its bytes. SHA-256 produces a 32-byte digest, written as 64 hexadecimal characters, and a store that keys artifacts this way gets three properties for free: identical content collapses to one entry, the name is verifiable by recomputation, and the name cannot be reused for different content without a collision.

The properties stop at the boundary of the object that was hashed. Hashing the checkpoint file gives a stable name for the tensors and nothing else. Two deployments that share that name can differ in the tokenizer revision, and a rollback that restores the checkpoint by digest will restore the tensors while leaving the current tokenizer in place. The result is a hybrid that never existed in any evaluation.

The fix is to hash a manifest rather than a blob. The manifest lists a digest per component and includes the runtime configuration as canonicalised data, and the deployment is named by the digest of the manifest. Because each component is itself content-addressed, the manifest is small: a few hundred bytes of digests and a configuration object. The components can be stored separately, cached by digest, and deduplicated across deployments that share a tokenizer.

import hashlib, json

def digest(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()

COMPONENTS = ("weights", "tokenizer", "preprocess", "runtime")

def identity(manifest):
    # Root covers each component digest plus the config that shapes behaviour.
    parts = [f"{name}:{manifest['components'][name]['digest']}" for name in COMPONENTS]
    parts.append("config:" + json.dumps(manifest["config"], sort_keys=True))
    return hashlib.sha256("\n".join(parts).encode()).hexdigest()

Canonicalisation matters more than it appears. Two configuration objects that mean the same thing but serialise with different key order or different numeric formatting produce different roots, so the digest of a semantically identical deployment changes. Sorting keys and fixing the number format removes most of that noise. What remains is the honest kind: a value that differs is a value that differs.

Divergences that survive a weight hash

The cases that cause incidents are the ones where the weights are correct and something adjacent is not.

Tokenizer revision. Tokenizer files are frequently fetched by name from a hub rather than pinned by digest. A repository that adds a special token or adjusts a normalisation rule keeps the same name, and a server that resolves the name at startup picks up the new file. Prompt length changes, the model sees ids it was not trained on, and outputs degrade in a way that no weight comparison detects.

Template drift. The chat template is a string interpreted at request time. Changing a role marker, a separator, or the presence of a generation prompt shifts every prompt by a few tokens. Instruction-tuned models are sensitive to this; the failure looks like a quality regression with no numerical cause.

Precision conversion. Weights stored in 32-bit floating point can be converted to 16-bit at load time. The conversion is deterministic for a given routine and format, but changing the routine, the format, or the accumulation dtype changes the logits. Two deployments that share the checkpoint digest and differ in conversion settings are different deployments, and the manifest has to record the conversion as part of the runtime configuration rather than as an implementation detail.

Serving limits. A maximum sequence length, a maximum batch size, or a truncation side (left versus right) changes which requests are refused and which are silently shortened. These live in server configuration and are exactly the kind of value that gets adjusted during a capacity incident and never restored.

Post-processing. A confidence threshold, a label map, or a decoding constraint applied after the model is part of the behaviour. A threshold change can move a classifier’s decision boundary across an entire population while the model stays fixed.

Each of these is a one-line change in a file that is not the checkpoint, and each is invisible to a digest of the checkpoint. The registry entry has to include them, which means the deployment pipeline has to construct the manifest from resolved values rather than from names.

Making the identity checkable at runtime

An identity that exists only in a registry is a claim about the past. The useful property is that a running server can prove which tuple it is executing, and that a load balancer or orchestrator can refuse to route to a server that cannot.

The mechanism is to have the server compute its own identity at startup from what it actually loaded, not from what it was told to load. The weights digest is computed over the bytes read, the tokenizer digest over the files parsed, and the configuration digest over the effective values after defaults are applied. The resulting root is exposed on a status endpoint alongside the registry digest. A mismatch between the two is a deployment error that can be caught before the server receives traffic, and a mismatch across servers behind one service name is a partial rollout that the control plane can detect.

Applying defaults before hashing is the part that is easy to get wrong. If the server hashes the configuration object as supplied and then fills in defaults at use time, two servers with different default tables produce the same digest and different behaviour. The effective configuration, after merging, is the only one worth hashing.

Why this belongs before rollout machinery

Rollout mechanisms assume a deployable unit that can be identified, compared, and restored. Shadow comparison needs to know whether the candidate differs from the baseline in weights alone or in preprocessing too, because a difference in preprocessing explains output differences that would otherwise be read as a model regression. Canary promotion needs a stable name for the thing being promoted, and the decision rule it applies is described in Canary Promotion as a Repeated Statistical Test. Rollback needs a name for the thing being restored, and restoring the wrong tuple is worse than not rolling back, since it produces a configuration that has never been evaluated; the drain and ordering problems that follow are covered in Rolling Back a Stateful Serving Fleet.

The cost of the tuple is small: a manifest of a few hundred bytes, one digest computation per component at build time, and one at server startup. The cost of skipping it is paid in incidents where the weights are innocent and the actual change is a template string that nobody recorded. Registries that store only checkpoint digests push that reconstruction into the postmortem, which is the most expensive place to do it.