Model Registries and Promotion Gates Between Training and Traffic

A model file on a shared filesystem with a timestamp in its name is a version. A model registry is what turns a version into an artifact with a lifecycle: an immutable identity, a recorded provenance, a stage that only advances on evidence, and a serving path that can name exactly one artifact at a time and change which one without rebuilding.

Identity, not naming

The first decision is what the identity of a model is. Three candidates compete, and only the third is stable enough for the rest of the system to depend on.

A name plus version number is what registries present. It is human-readable and mutable at the edges: a version number can be reassigned, a name can point to different artifacts in different environments, and nothing prevents two runs from registering the same version.

A model family is a set of versions that share a training pipeline definition and an interface: same input feature set, same output semantics. The family is the unit of compatibility: a consumer that resolves a family can accept any member, subject to the schema check below.

A content hash of the packaged artifact is the immutable identity. Two registrations with the same hash are the same artifact, and the hash is what the serving log records. Version numbers become a display convention over it, and a promotion moves a pointer rather than rewriting a file.

The registry stores all three, but only the hash is trustworthy for audit. A serving log that records “model v17” answers the version question only if v17 was never reassigned, which is a property no filesystem guarantees.

The provenance that belongs in the record

The metadata attached to an artifact is what makes it explainable months later. Four groups belong there, and each has a distinct consumer.

The training record holds the data snapshots, code commit, hyperparameters, seeds, and environment, recorded as described in reproducing training runs from metadata. Its consumer is the person investigating a regression.

The feature contract holds the feature hashes and the serving-side computation the model expects. Its consumer is the serving path, which must fail at load time if the feature computation it would use does not match, rather than silently feeding the model a different distribution. This is the mechanism that prevents the skew described in training-serving skew from becoming a runtime failure of the model rather than a check at startup.

The evaluation results record the metrics on the fixed evaluation snapshot, with the snapshot identity and the confidence intervals. Its consumer is the promotion gate.

The package holds the serialized weights plus everything needed to reproduce the forward pass: the preprocessing, the tokenizer or vocabulary, the library version that deserializes the weights. A weights file alone is not an artifact; the code that interprets it is part of the model’s definition, and a registry that stores only weights leaves serving to reimplement the interpretation.

Stages and the evidence each transition requires

Registries commonly expose stages with names such as staging and production, and the valuable part is not the label but the rule for entering it. A transition without a rule is a comment.

Training to staging should require only that the artifact loaded and produced the recorded metrics when scored on the evaluation snapshot. It is a smoke test, and its purpose is to reject artifacts whose package is broken. This transition should be cheap enough that every candidate passes through it.

Staging to production is the gate that matters. Four checks recur across systems. First, the candidate beats the incumbent on a common evaluation set, by a margin derived from the metric’s standard error rather than a fixed epsilon. Second, the input schema of the candidate matches the schema the serving path supplies, including nullability and the treatment of missing values. Third, a shadow or canary period has measured the candidate’s latency and error rate on real traffic, since these are invisible offline. Fourth, the rollback target is identified and its own package is verified loadable, because a rollback to a broken artifact is worse than no rollback.

The ordering is deliberate: cheap checks before expensive ones, and the rollback verification before any traffic is shifted. Systems that verify the rollback after the promotion have a window in which they can neither move forward nor back.

Production to archived should be automatic after a retention period, and the retention should be long enough that every artifact ever served can still be named by the hash in a log. Deleting an artifact that served traffic converts an audit question into an unanswerable one.

Resolving a version at request time

How the serving path picks a version determines what a promotion means operationally. Three designs are common and they differ in where the cost is paid.

A version baked into the deploy is simple and fully controlled: promotion means a new deployment, with whatever rolling process the platform provides. It couples model change to code change, which is appropriate when the model’s forward pass is compiled into the service and inconvenient when only the weights changed.

A version resolved at startup from the registry decouples the two: the service reads a pointer, loads the artifact, and keeps it for its lifetime. Promotion then requires a restart or a reload signal. The failure mode is a startup dependency on the registry, which turns a registry outage into an inability to start new replicas.

A version resolved per request, or per interval, allows the fastest promotion and the fastest rollback, at the cost of hot reloading the artifact and the risk of serving two versions within one response. Where the model is small and reload is cheap, this is workable; where loading takes seconds and memory is significant, it requires keeping both versions resident.

Whichever design is chosen, the served version must be logged with the prediction. Without that, the registry describes intent and the logs cannot describe reality.

Immutability and the failure it prevents

The registry’s central guarantee is that an artifact’s bytes do not change after registration. This sounds trivially enforceable and is routinely violated in three ways.

Mutable storage behind the pointer. An artifact referenced by a path into a bucket that anyone can overwrite is mutable in practice. Content-addressed storage, where the path is derived from the hash, makes an overwrite impossible without changing the identity.

Deserialization drift. The same weights file loaded by a newer library version can produce different predictions, because a layer’s default changed or a numerical path differs. The registry’s record of the deserialization version is what makes this detectable; without it the artifact is immutable and its behaviour is not.

Unpinned transitive dependencies. A package that pins the training library but not the numerical library beneath it inherits whatever version the index resolves on the day it is deployed. Pinning the full dependency set, as container images by digest do, closes this.

Each violation produces the same class of incident: a model whose offline and recorded evaluation no longer describes what production runs. The registry cannot prevent the incident, but it can make it observable, which is the difference between a diagnosis and a mystery.

Multi-armed traffic and the meaning of the incumbent

Once traffic is split between versions, the registry’s notion of “the production version” becomes a distribution rather than a single artifact. Two consequences follow. Rollback must be defined in terms of the traffic allocation, not only the pointer: reverting the pointer while a percentage of requests still routes to the candidate leaves the candidate serving. And evaluation of the split must account for the allocation: the two arms see different traffic only if assignment is random, and if assignment is by entity, the same user always gets the same version, the comparison is between two populations and requires the same care as the offline split design discussed in offline evaluation leakage.

The recorded allocation over time is part of the model’s operational history, and it is the only way to reconstruct which version produced a given prediction when the logs carry a request identifier rather than a version. Registries that record only stage transitions lose that history at exactly the granularity where it is needed.