Autoscaling a Token Service with Cold Starts and Diurnal Load

Reactive autoscaling works when the resource being added is available in seconds. For a token service it is available in minutes, and the delay is structural: a new replica must be scheduled onto hardware, the container image must be pulled, the weights must be read from storage into accelerator memory, the kernels must be compiled or loaded from cache, and the process must pass a readiness check before it accepts traffic. A policy that reacts to observed load therefore adds capacity after the load has already arrived, which is the moment when queueing is worst.

The economics of the problem are a comparison between two ways of paying for the same readiness. A warm pool pays continuously for replicas that are idle most of the time. Predictive scaling pays nothing for idle replicas but accepts a risk of being late, and the cost of being late is queueing delay or dropped requests. Which is cheaper depends on the lead time, the sharpness of the load curve, and the price of a violated latency target.

The lead time, decomposed

Cold-start latency is the sum of stages that can each be attacked separately. Scheduling onto a node is fast when capacity exists and unbounded when it does not, since a pending pod waits for another workload to release hardware. Image pull is proportional to image size and network bandwidth; a multi-gigabyte image over a constrained link is a minutes-scale operation. Weight loading is proportional to model size and storage bandwidth: a 140 GB checkpoint read at a few gigabytes per second takes tens of seconds at best, and longer if the storage is remote and shared. Kernel compilation, where it happens at first use, adds seconds to minutes depending on the framework’s caching.

The sum is the reason a replica cannot be treated as an instantly available unit. It also shows which stage dominates: for large models it is weight loading, for small models it is often image pull and scheduling. Reducing the dominant stage is the only intervention that shortens the lead time materially; optimizing the others changes the total by seconds.

Caching weights on the node removes the storage read for any replica scheduled there, at the cost of node-local storage sized for the models the node may serve. A node that keeps a model resident can start a replica in the time it takes to initialize the runtime rather than the time it takes to read the checkpoint. This is the same trade as a warm pool but at a different granularity: the weights are warm rather than the replica, and the saving is the loading stage only.

Why a utilization target lags

A scaling rule of the form “add a replica when mean utilization exceeds 0.7 for two minutes” is a feedback controller with a delay in the plant. The delay is the cold-start time plus the measurement window plus the time for the new replica’s effect to appear in the metric. During that interval, demand continues to arrive at the higher rate and the existing replicas absorb it, so their utilization rises further, which triggers more scale-out, which arrives later still.

The result is oscillation at best and a queue collapse at worst. The collapse happens when the delay is comparable to the duration of the peak: capacity arrives as the peak is ending, is measured as excess, and is removed just before the next one. The controller then spends the whole cycle chasing a load curve it cannot see.

The standard fix is to feed the controller a signal that leads the load rather than one that reports it. Time-of-day patterns repeat, so the load curve of the previous week is a usable predictor for the current one, and capacity can be provisioned against the forecast rather than against the measurement. The forecast does not need to be accurate in the mean; it needs to be accurate in the timing of the ramp, because the ramp is what the lead time must cover. A forecast that is early by ten minutes costs ten minutes of idle capacity; one that is late by ten minutes costs ten minutes of queueing.

Sizing the warm floor

Between fully predictive and fully reactive sits a warm floor: a number of replicas kept resident at all times, sized to the minimum of the load curve rather than to its mean. The floor exists to absorb the lead time at the start of a ramp, so its size is set by how much load arrives during one lead time at the steepest part of the curve.

If the load rises at rate $r$ requests per second per second and the lead time is $T$ seconds, the extra load arriving during the lead time is $rT$ requests per second. Those requests must be served by the floor until the elastic capacity lands, so the floor must cover the load at the start of the ramp plus $rT$. For a ramp that doubles over an hour and a lead time of several minutes, $rT$ is a meaningful fraction of the ramp’s total, and the floor is correspondingly larger than the curve’s minimum.

The alternative to a larger floor is a shorter lead time, and the two are directly exchangeable. Halving the cold start halves the required floor at a given ramp rate. This is why weight caching, pre-pulled images, and reserved nodes are economically interesting even when they cost storage: they buy a smaller permanent floor.

def warm_floor(base_rps, ramp_rps_per_s, lead_time_s,
               rps_per_replica, headroom=1.3):
    # capacity that must be resident when the ramp begins
    needed_at_ramp_start = base_rps + ramp_rps_per_s * lead_time_s
    return headroom * needed_at_ramp_start / rps_per_replica

# headroom covers the utilization target from the queueing calculation;
# it is not a safety factor on top of an already-utilized pool

The headroom term here is the same one derived in the replica sizing analysis, applied to the floor rather than to the whole pool. Applying it twice, once to the floor and once to the elastic capacity, is a common error that inflates the plan by the square of the factor.

The cost of a warm replica, and of a missing one

A warm replica costs its hourly rate for every hour it is resident, which for a diurnal workload is most of the day. A cold replica costs nothing until it is needed. The comparison therefore turns on how many hours of the day the warm replica’s capacity is actually required, which is the reserved-capacity break-even question again, with the discount replaced by the cost of the alternative.

The missing replica has a cost too, and it is easier to underestimate. A request that waits beyond its latency target is a degraded response; a request that is rejected is a lost one. If the service has a queue with a bounded length, excess load is dropped, and the dropped requests are the visible cost. If the queue is unbounded, the cost is delay, which spreads across all requests and produces timeouts at the client, which produce retries, which add load. The feedback from retries is the dangerous part: a pool that is slightly short can be pushed far short by its own clients’ retry behavior.

Quantifying the cost of a missing replica requires a value per request and a distribution of excess load, which most capacity plans do not have. In their absence the defensible approach is to treat the latency target as a hard constraint and size the floor to meet it at the forecast peak ramp, then measure the resulting idle fraction and decide whether the floor is worth its cost. That inverts the usual framing: instead of asking how much latency degradation is acceptable to save capacity, it asks how much idle capacity is worth avoiding a violation.

Scale-in, and the asymmetry with scale-out

Scale-out is slow and scale-in is fast, and the policies should not be symmetric. Removing a replica is a matter of draining it: stop admitting new sequences, let in-flight ones finish or migrate them, then terminate. Draining takes as long as the longest request in flight, which for a streaming service can be minutes. A scale-in policy that acts on a short window will remove capacity that is about to be needed again, and the next scale-out will pay the full cold start.

The asymmetry suggests hysteresis: a longer window and a lower threshold for scale-in than for scale-out, so that capacity is removed only after the load has been below the threshold for long enough to distinguish a lull from a fluctuation. The cost of the hysteresis is idle replicas during the window; the benefit is avoiding a cold start. For a workload with a predictable diurnal shape, the window can be set from the shape itself rather than from a generic default.

A second asymmetry is that scale-in must respect the memory of in-flight work. A replica holding long-context sequences cannot be drained quickly, and forcing it would waste the prefill already performed. Draining policies that wait for sequences to complete naturally are cheap in tokens and expensive in time; those that preempt are the reverse. The choice depends on whether the prefill can be recovered from a prompt cache, which turns an expensive drain into a cheap one when the prefix is shared.

What remains unpredictable

Forecasting handles the diurnal component and fails on the rest. A product launch, a traffic shift from another region, or a client retry storm are not in last week’s curve, and the lead time means the response to them is late by construction. The only defenses are a warm floor sized for more than the forecast minimum and admission control that degrades gracefully rather than collapsing: queueing with a deadline, shedding the least valuable requests first, and returning an explicit overload signal so clients can back off instead of retrying.

Those mechanisms are not autoscaling, but they are what makes autoscaling safe. A pool that scales perfectly and has no admission control still fails when demand exceeds the forecast; a pool that scales imperfectly and sheds load predictably stays within its latency target at a lower capacity. The capacity plan should specify both, because the second determines how much the first is allowed to be wrong.