Autoscaling Model Servers on Queue Depth Rather Than Utilization

The default autoscaling signal is CPU utilization: keep average utilization near a target, add replicas when it rises, remove them when it falls. For a request-response service whose handler is a few milliseconds of interruptible work, this is adequate. For a model server it is not, because the work per request is long relative to the control interval, the relationship between utilization and latency is sharply non-linear near saturation, and a replica that is removed cannot finish the requests it was holding.

The result of applying the default signal is a control loop that oscillates: it adds capacity after latency has already degraded, removes it while the new replicas are still warming, and re-adds it. The fix is to choose a signal that predicts the thing being protected rather than a proxy that correlates with it only in the linear region.

Why utilization saturates before latency does

For a single-server queue with arrival rate $\lambda$ and service rate $\mu$, utilization is $\rho = \lambda/\mu$, and the expected wait in the queue for a memoryless arrival and service process is $\rho/(1-\rho)$ times the mean service time. At $\rho = 0.5$ the wait is one service time; at $\rho = 0.9$ it is nine; at $\rho = 0.95$ it is nineteen.

The curve is flat until it is vertical. A utilization target of 0.7 sits in the region where the derivative is still moderate, and a burst that pushes the arrival rate up by 30% moves the wait by a factor of three. The autoscaler, seeing 0.7 rise to 0.9, needs a control interval plus a replica start-up to respond, and by then the queue has absorbed the burst and the wait is at its peak.

Model servers make this worse in two specific ways. The first is that a replica’s start-up is not milliseconds: loading weights, initialising kernels, and warming caches can take tens of seconds to minutes depending on model size and storage bandwidth. During that window the new replica contributes nothing and may consume shared bandwidth. The second is that a model server’s throughput is not fixed: batching means a replica’s service rate depends on its batch composition, so $\mu$ is itself a function of the load.

The consequence is that utilization lags the thing it is supposed to predict by exactly the interval in which the prediction would have been useful.

The signal that matches the SLO

The quantity a user experiences is the time from arrival to first token, and the part of it that capacity controls is the wait before service begins. That wait is observable directly: at admission, the server knows how long a request has been queued, and it knows how many requests are waiting.

Queue depth is therefore a better signal than utilization, and it has a property utilization lacks: it is measured in the same units as the harm. A target of “no request waits more than 200 ms before admission” translates into a queue-depth target through Little’s law, $L = \lambda W$, where $L$ is the mean number in the system, $\lambda$ the arrival rate, and $W$ the mean time in system. At 200 requests per second and a 200 ms wait target, the queue should hold about 40 requests. That number is a direct, checkable target, and it moves with traffic in a way that utilization does not.

Queue depth alone is still a proxy, because the wait also depends on service time. A more direct signal is the age of the oldest waiting request, which is bounded above by the wait any admitted request will experience. If the oldest waiting request is 500 ms old and the target is 200 ms, the fleet is over target regardless of how many requests are queued or how busy the replicas are.

def desired_replicas(oldest_wait_s, target_wait_s, current, min_r, max_r):
    # Multiplicative control on the ratio of observed to target wait.
    if oldest_wait_s <= 0:
        return current
    ratio = oldest_wait_s / target_wait_s
    want = math.ceil(current * ratio)
    return max(min_r, min(max_r, want))

The multiplicative form is chosen because the relationship between wait and required capacity is roughly proportional in the region of interest: doubling the wait means roughly doubling the service capacity needed to clear it. A control law that adds a fixed number of replicas per interval cannot keep up with a burst that arrives ten times faster than the previous interval, and a law that doubles can overshoot badly when the wait is already high. Damping the multiplier, and capping the change per interval, are the standard remedies.

Hysteresis, and why scale-down must be slower than scale-up

A control loop that acts on both directions at the same rate spends its time adding and removing replicas around the target. Two mechanisms produce the oscillation.

The first is measurement noise. Queue depth at a one-second resolution fluctuates; a rule that reacts to every fluctuation issues a scaling decision per fluctuation. Smoothing the signal over several intervals, and requiring the smoothed value to exceed the threshold for more than one interval, removes most of this.

The second is the asymmetry of cost. Scaling up too late costs latency for the duration of the delay. Scaling down too early costs a cold start on the next burst, which is longer than the scale-up delay because the replica has to reload. The two errors are not symmetric, so the thresholds should not be either: a common arrangement is to scale up at a threshold and scale down only below a noticeably lower threshold, sustained for several intervals.

A concrete form: add replicas when the oldest wait exceeds the target, remove one replica per interval when the oldest wait is below half the target and utilization is below a lower bound, and never remove below a floor that covers the observed diurnal minimum. The floor is what prevents the loop from removing capacity during a lull and then being unable to restore it before the next peak.

Warm capacity and the replica that is not ready

A replica that has been created is not a replica that can serve. The distinction is formalised as readiness: the orchestrator routes traffic only to replicas that report ready, and the readiness check should not pass until the model is loaded and a warm-up request has completed.

Three properties follow from taking readiness seriously.

Warm-up must exercise the actual serving path. A readiness check that returns a static response confirms the process is alive and says nothing about whether the model is loaded. A check that runs a real inference confirms the whole path, at the cost of consuming capacity on every check.

The autoscaler must count only ready replicas when computing the desired count. Counting pending replicas as capacity makes the loop believe it has already responded, and it stops adding while the pending replicas are still minutes away from serving.

Scale-down must drain rather than delete. A replica removed from the desired set still holds in-flight requests, and terminating it truncates them. The drain procedure described in Rolling Back a Stateful Serving Fleet applies here unchanged: remove from routing, wait for propagation, finish in-flight work, then exit.

Pre-warming and the cost of a fast reaction

The cheapest way to handle a burst is to have capacity before it arrives. Predictive scaling, which schedules capacity from a historical pattern, handles diurnal cycles well and unpredictable events not at all.

Reactive scaling handles the unpredictable case, bounded by start-up time. When start-up is tens of seconds, a reactive loop cannot protect a latency SLO during a step change; it can only bound the duration of the degradation. The mitigation is to keep a pool of warm replicas that are ready but not routed to, so that a scale-up event promotes existing capacity rather than creating it. The pool costs the idle memory and compute of running a model without serving, which for a large model is the dominant cost.

The trade is explicit: idle warm capacity is paid for continuously and consumed only during bursts; cold capacity is paid for only when used and unavailable exactly when needed. The size of the pool is a function of how much latency degradation is acceptable during a burst and how long a cold start takes, and both of those are properties of the specific deployment rather than general rules.

Where the loop interacts with rollout

An autoscaler and a rollout controller both change the replica set, and they can disagree. A rollout that is replacing replicas one at a time reduces capacity temporarily, and an autoscaler reacting to the resulting queue growth adds replicas, some of which run the new version and some the old. The result is a rollout that never converges because the desired count keeps changing under it.

The usual resolution is to suspend autoscaling during a rollout, or to give the rollout controller a fixed replica budget and let the autoscaler act only within it. Neither is free: suspending autoscaling during a rollout leaves the service unable to absorb a burst at exactly the moment when capacity is already reduced.

A more careful arrangement keeps the autoscaler running but makes the rollout a constraint rather than a competing controller: the autoscaler computes a desired total, the rollout controller decides the version split within that total, and the two never write the same field. That requires the deployment to express the split as a separate object from the count, which is the property that makes the interaction tractable. Where the two controllers do write the same field, the symptom is a rollout that appears stuck at a partial split while the total replica count climbs, and the cause is that each controller is reacting to the capacity the other removed.

The last interaction is with the rollback path. An automatic rollback that scales the fleet down to the previous version’s replica count, while the autoscaler is still reacting to a queue that grew during the incident, produces a double action: the rollback removes capacity and the autoscaler adds it back, and the resulting count is neither what the previous version ran with nor what the current load requires. Latching the rollback and letting the autoscaler own the count afterwards avoids it, and the latching mechanism is described in Automatic Rollback Controllers.