Inference Systems · Cluster
ML Operations
Deployment, monitoring, and rollback.
- 01 Model Artifact Identity: Why a Checkpoint Hash Is Not a Deployment The behaviour a serving fleet exhibits depends on far more than tensor values, so deployment identity has to cover the tokenizer, the preprocessing path, and the runtime configuration as one unit.
- 02 Shadow Traffic: Duplicating Requests to a Candidate Without Serving Them How a serving stack runs a candidate model on a copy of live requests, what the duplication costs at the proxy, and why the comparison it produces is weaker evidence than it looks.
- 03 Canary Promotion as a Repeated Statistical Test A canary compares a small exposed fraction against the incumbent, which means the promotion rule is a hypothesis test run many times, and its false-positive rate depends on how often it is evaluated.
- 04 Rolling Back a Stateful Serving Fleet Under Live Traffic Restoring an earlier model is simple until the serving fleet holds request state, warm caches, and in-flight streams; then the rollback itself becomes a state transition with its own failure modes.
- 05 Point-in-Time Feature Reads and the Skew They Prevent Training a model on features computed after the fact and serving it on features computed at request time yields two different distributions; point-in-time joins are the mechanism that forces them to agree.
- 06 Where Latency Is Measured, and Why Percentiles Do Not Average A serving latency number is a function of the point where the clock starts, and percentiles computed at different layers cannot be combined; both facts determine whether a monitoring figure means anything.
- 07 Windowed Drift Detection and the Arithmetic of Its Alarms Comparing a live feature distribution against a reference is a two-sample test on a window, and the window length, the binning, and the number of features being tested together determine how often the alarm fires with nothing wrong.
- 08 Replaying Logged Traffic to Evaluate a Candidate Offline Recording request and response pairs makes it possible to score a new model against decisions that were already made, but replay is only valid when the candidate sees the same context the incumbent saw.
- 09 Autoscaling Model Servers on Queue Depth Rather Than Utilization Utilization-based autoscaling assumes work is interruptible and cheap to queue; for inference it produces oscillation, so the control signal has to be the wait time a request would experience.
- 10 Automatic Rollback Controllers and the Ways They Misfire A controller that reverts a deployment on a metric threshold is a feedback loop wrapped around a feedback loop; its failure modes are oscillation, latency in the evidence, and disagreement about what baseline means.