Rolling Back a Stateful Serving Fleet Under Live Traffic

Rolling forward is a decision made once. Rolling back is a decision made under time pressure, against a fleet that is actively serving, and the state that has accumulated since the rollout began has to be dealt with rather than ignored. A model server that holds a decoded token stream, a prefix cache built from the new version’s tokenizer, or a connection pool sized for the new version’s concurrency cannot be replaced by swapping a pointer.

The mechanics differ by how much state the serving path owns. A stateless classifier behind a queue is nearly trivial to revert. A generative serving fleet with paged key-value caches, streaming responses, and per-session affinity is not, and the rollback procedure has to be designed before it is needed.

The four kinds of state a rollback must handle

Separating them clarifies which need action and which can be discarded.

In-flight requests. A request that has been admitted and is partway through generation exists only in the memory of a specific server process. Killing the process drops it. The user sees a truncated stream or a dropped connection, and the client may retry, producing a duplicate generation.

Response streams already sent. A client that has received 200 tokens of a 500-token answer cannot be un-sent those tokens. If the rollback is because the outputs are wrong, the tokens already delivered are wrong and the client has no way to know.

Caches keyed to the new version. A prefix cache, a prompt cache, or a compiled kernel cache built against the new weights or the new tokenizer is not valid for the old version. Reusing it produces either a correctness failure, if the key does not encode the version, or a miss, if it does. The second is safe and slow; the first is a silent bug.

External side effects. A model that writes to a database, enqueues work, or updates a user’s record has already committed those effects. Rolling back the model does not roll back the effects, and the old model may then produce inconsistent follow-on actions.

Only the first is handled by the standard deployment machinery. The other three are the reasons a rollback is not symmetric with a rollout, and the cache case is the one that shows why the deployment identity has to include the version: a cache whose key omits it silently serves stale state, which is argued in Model Artifact Identity.

Draining without truncating

The default termination behaviour of an orchestrator is to send a signal and wait a fixed grace period before killing. For a stateless HTTP handler that is adequate. For a generative server it is not, because generation time is proportional to output length and the tail of that distribution extends well past any fixed grace period.

The correct sequence has four steps, and each exists for a specific failure.

1. Remove the instance from the routing set. The load balancer stops sending new requests. This must happen before anything else, and it must be confirmed at the load balancer, not merely requested. A controller that stops routing and a load balancer that has applied the change are separated by a propagation delay, during which requests continue to arrive.

2. Wait for the propagation delay. A conservative wait, on the order of the load balancer’s configuration refresh interval, covers the window in which new requests are still being routed. Skipping this step produces connections that arrive after the process has begun shutting down, which surface as connection resets at the client.

3. Signal the process to stop accepting and finish what it holds. The server stops admitting, completes in-flight work, and exits when its in-flight count reaches zero or a hard deadline passes.

4. Force-terminate at the deadline. Streams that have not finished by the deadline are cut. The deadline exists because some requests will never finish: a client that stops reading, a generation that hits no terminator, a downstream call that hangs.

The deadline is the interesting parameter. Setting it short truncates legitimate long generations and converts a rollback into an error spike. Setting it long makes the rollback take as long as the slowest request, which for a p99 generation time can be tens of seconds, and the fleet runs at reduced capacity for that entire period.

// Drain: stop accepting, let in-flight work finish, then force exit.
func (s *Server) Drain(ctx context.Context, grace time.Duration) error {
    s.accepting.Store(false)
    deadline := time.Now().Add(grace)
    for {
        if s.inflight.Load() == 0 {
            return nil // clean exit
        }
        if time.Now().After(deadline) {
            s.cancelAll() // cut remaining streams
            return errDeadlineExceeded
        }
        time.Sleep(100 * time.Millisecond)
    }
}

The loop polls rather than waiting on a condition variable because in-flight counts change from many goroutines and a poll at 100 ms granularity is cheaper than the coordination. The exit path is the same either way; what matters is that the deadline is enforced by the server rather than left to the orchestrator’s kill timer, so that the server can cancel its own work and release resources in order.

Capacity during the transition

A rolling rollback replaces instances gradually, which means that for the duration of the transition the fleet is a mixture of two versions. That mixture is not free.

The first cost is capacity. If the old version has a larger memory footprint per replica, replacing a new-version replica with an old-version replica reduces the number of concurrent sequences the node can hold. A rollback triggered by a memory-related failure can therefore be slowed or defeated by the resource profile of the version being restored.

The second cost is heterogeneous output. During the transition, the same input routed to different replicas produces different answers. For a stateless request-response API this is invisible. For a multi-turn conversation where the client sends the previous turn back, it is visible: turn three may be generated by the old model from a history produced by the new one. Whether that matters depends on the model, and it is not something the rollback machinery can determine.

The third cost is cache invalidation. If caches are keyed by version, the transition produces a cold cache for each replica as it comes up, which concentrates load on the backend that populates it.

A rollback that must be fast trades these away by replacing the whole fleet at once, accepting a brief period of zero capacity in exchange for a short total duration. A rollback that must preserve availability rolls gradually and accepts the mixture. The choice is a property of the incident, which is why the procedure needs both modes available and a rule for selecting between them.

Why rollback targets must be immutable

The common failure in rollback systems is not the drain logic. It is that the artifact the rollback points to has changed since the rollout began.

If the rollback target is a mutable tag, such as a name that the build system reassigns on each push, then rolling back to that tag restores whatever it points to now, which may be a version that was never in production. If the target is a branch, the same problem. Only a digest or an immutable version number identifies the bytes that were actually running.

The second failure is that the rollback target’s dependencies have moved. A deployment identity that includes the tokenizer and the runtime configuration can only be restored if those components are also retained. A registry that garbage-collects unreferenced artifacts after some retention window will eventually delete the exact components needed to restore an older deployment, and the deletion is invisible until the rollback is attempted.

Retention policy is therefore part of rollback readiness. The set of deployments that must remain restorable is not all deployments; it is the current one plus every one that could plausibly be restored during an incident, which in practice means the last few versions plus anything currently referenced by a rollback plan.

Ordering: model first or configuration first

A rollback often involves both a model and a configuration change, because the two were deployed together. The order in which they are reverted determines the intermediate state, and the intermediate state may be worse than either endpoint.

Reverting the configuration first, with the new model still running, produces the new model under the old configuration. If the configuration controls sampling parameters or safety thresholds, that combination may be untested. Reverting the model first, with the new configuration still applied, produces the old model under the new configuration, which is equally untested.

The resolution is to make the deployment unit atomic: the model and its configuration are one artifact with one identity, and the rollback swaps the whole unit. This is the practical argument for the manifest-based identity described in Model Artifact Identity. A rollback that can only address the weights has to reason about which half of the change to revert first; a rollback that addresses the tuple has one operation and no intermediate state.

Verification that the rollback worked

A rollback is complete when traffic is being served by the intended version, not when the controller reports success. The gap between those two is where partial rollbacks live: a controller that updated three of five deployments, a load balancer that kept routing to a stale endpoint, a replica that failed to start and was replaced by one running the previous version because the image tag resolved differently.

The check is the runtime identity comparison: each serving instance reports the identity it computed from what it loaded, and the control plane compares that set against the intended identity. A rollback is verified when every instance in the routing set reports the target identity and the routing set is the expected size. Anything else is an incomplete rollback, and the incident is not over.

This is also the point at which the post-incident work starts. The rollback restored service; it did not restore the reason the rollout was attempted. The next attempt needs the failure to be diagnosed, and the evidence for that diagnosis is the metrics collected during the canary window and the outputs recorded from the failing version. A rollback procedure that also preserves that evidence, by snapshotting the failing version’s logs and metric window before the fleet is replaced, saves the reconstruction later.