Autoscaling Signals for Token Servers

An autoscaler maps an observed signal to a replica count. The mapping is only as good as the signal’s relationship to saturation, and for token generation the conventional signals are close to useless. Accelerator utilization sits near its maximum over a wide range of loads, because the decode loop keeps issuing work regardless of whether that work is useful. Request rate says nothing about how much work each request represents. A signal that is flat across the region where scaling decisions matter cannot drive those decisions.

Why the obvious signals fail

Utilization is the clearest failure. In a decode iteration, the accelerator is busy whether the batch holds four sequences or forty. The difference between those states is throughput per unit time and latency per sequence, not idle time. A utilization target of, say, seventy percent is reached at low load and stays reached well past the point where latency targets break. An autoscaler acting on it adds replicas too late, or never.

Request rate fails differently. One request may carry a 50-token prompt and produce 20 tokens; another may carry a 30,000-token prompt and produce 2,000. The second is three orders of magnitude more work. A fleet scaled on requests per second treats them as equal, so a shift in the workload’s mix produces saturation with no change in the signal that drives scaling. The same objection applies to active connection count, which counts idle keep-alives as load.

Time per request fails because it conflates the queue with the service. A request’s total latency is waiting time plus generation time, and generation time grows with output length while waiting time grows with backlog. A p95 that rises because outputs got longer does not indicate a capacity shortage, and scaling on it adds replicas that will not shorten the outputs.

Signals that track work

Three signals measure the quantity that actually saturates.

Queue depth in tokens is the first. For prefill capacity, the relevant figure is the number of prompt tokens waiting for their first forward pass. For decode capacity, it is the number of resident sequences times their expected remaining output, which the server cannot know exactly but can bound with the client’s requested maximum. Either way the unit is positions, not requests, and the value is additive across heterogeneous requests in a way that request count is not.

Cache occupancy is the second. Because the batch that runs is limited by available blocks, the fraction of blocks in use is a direct measure of how close the replica is to refusing admission. At low occupancy, added sequences cost latency but not capacity; near full occupancy, the next admission forces a preemption decision and the latency of every resident sequence rises. Occupancy above roughly eighty percent is the region where the queueing behaviour becomes non-linear, and it is a better scaling trigger than any utilization figure.

Time-to-first-token is the third, and it is the only one that reflects the client’s experience of the prefill path. It is also the signal that responds fastest to a burst of long prompts, since those enter the prefill queue and immediately raise the wait for everyone behind them.

A workable policy combines them: scale out when cache occupancy is high and time-to-first-token exceeds its target, scale in when occupancy is low across a sustained window. The conjunction matters, because either condition alone produces false positives. High occupancy with a fast first token is a replica that is efficiently full; a slow first token with low occupancy is a replica whose prefill share is too small, which is a configuration problem rather than a capacity problem.

The lag that makes reactive scaling insufficient

Autoscaling has a delay between decision and capacity: the time to schedule a pod, pull an image, load weights, initialize the runtime, and warm the cache. Loading a 14 GB model from network storage and initializing a runtime is not instant, and the first requests served by a cold replica are slower than steady state. The total delay is measured in tens of seconds to minutes depending on image size and storage, while a traffic burst can build in seconds.

Two consequences follow, and they are architectural rather than operational.

The first is that the fleet needs headroom sized to the burst, not to the mean. The correct amount is derived from how fast the workload can grow relative to the cold-start delay: if traffic can double in ten seconds and a replica takes a minute to become useful, capacity for the doubled rate must already exist. That headroom is a cost, and it is the reason serving fleets are often sized by peak-to-mean ratio rather than by utilization.

The second is that the router must protect the cold replica while it warms. Sending it a full share of traffic immediately produces timeouts that look like failures, and the natural response, retrying, makes it worse. Ramping traffic over a warm-up window, or holding the replica out of the pool until it reports readiness, converts a failure into a delay. The interaction with load scoring is covered in Least-Outstanding Routing.

# sketch of a scaling policy expressed on work, not on utilization
scaleOut:
  when: cacheOccupancy > 0.8 and ttftP95 > 1.5s
  for: 60s
  by: 25%
scaleIn:
  when: cacheOccupancy < 0.35 and ttftP95 < 0.4s
  for: 600s
  by: 10%

The asymmetry in the windows is deliberate. Scaling out must be fast because the cost of being short is client-visible failure; scaling in must be slow because the cost of being long is money, and because a premature scale-in that lands during a burst causes flapping.

Scaling the two phases separately

Once prefill and decode run on separate pools, one signal cannot serve both. Prefill load is measured in prompt tokens queued and in time-to-first-token; decode load is measured in resident sequences, cache occupancy, and inter-token latency. The two saturate independently, and a single replica count forces them to scale together, which wastes capacity on whichever pool is not the bottleneck.

The split also changes the meaning of headroom. Prefill is stateless between requests, so a prefill replica becomes useful as soon as its weights are loaded; a decode replica additionally needs its cache warmed by traffic, and its steady-state capacity is reached only after enough sequences are resident. The decode pool therefore needs a longer readiness window and a larger buffer against bursts.

There is a coupling to watch for. If decode capacity grows, prefill workers stop accumulating cache they cannot hand off, so their own throughput rises without any change to their count. Scaling one pool can therefore mask a shortage in the other, and the signals must be examined together rather than independently. The protocol-level reason is set out in Prefill and Decode Disaggregation.

What scaling cannot repair

A fleet can be at the correct size and still fail its latency targets if the admission policy lets too many sequences into each batch. The batch size that maximizes throughput and the batch size that meets an inter-token deadline are different numbers, and the difference is a policy choice described in Deadline-Aware Batching and Tenant Fairness. Adding replicas reduces the number of sequences per replica, which shortens each batch, so scaling does help, but only if the policy leaves room for the reduction to matter.

Similarly, a fleet can be correctly sized and still exhibit long tails if the router’s load signal is stale or if retries land on already-failing replicas. Scaling responds to aggregate demand; it does not correct a distribution problem within the fleet. The placement constraints that determine how many replicas can exist at all, and where they can sit relative to the interconnects they need, are the subject of Replica Packing and Topology Constraints.