Provisioning Lag and the Reaction Time of a Fleet

Capacity plans usually state a replica count for a peak. The number is a function of demand, and demand is not the only thing that arrives faster than capacity can. Between the moment a load increase becomes visible and the moment a new replica can accept traffic sits a chain of delays: the metric window that made the increase visible, the controller’s decision interval, the time to obtain a machine, the time to pull an image and start a process, and the time for the load balancer to consider the replica healthy. Each delay is a period during which the fleet is smaller than its target, so the plan must carry capacity for the highest load the system may see before a scaling action completes.

The control loop, decomposed

An autoscaler is a controller with a sampling period, a target, and an actuator. In the Kubernetes Horizontal Pod Autoscaler, the controller reads resource metrics at a configurable period, computes a desired replica count from the ratio of current to target utilization, and writes the new count to a workload object. The formula is a straightforward proportional controller: desired equals current replicas times the ratio of the observed metric to the target metric, rounded up.

The proportional shape has a consequence that catches people out. The controller does not react to the derivative of load; it reacts to the level. If load steps from half the target to twice the target, the computed desired count doubles in one pass only if the metric updates instantaneously, which it does not. If the metric is a rate averaged over a window, the observed value climbs as the window fills, and the controller adds replicas incrementally as it climbs. For a step increase in demand, the time to reach the correct count is therefore a function of the metric window, not of the load.

A second-order problem appears when the metric lags. If the observed metric is itself an average over a period longer than the controller interval, the controller keeps adding replicas after load has already fallen, which produces oscillation. That is why the configuration exposes a stabilization window for scale-down: it is a deliberate delay inserted to trade a little excess capacity for the absence of flapping. The asymmetry is justified, since excess capacity is merely expensive, but rapid downscale followed by upscale is destabilizing.

The delays that dominate

Ranking the components of provisioning lag is what makes the loop tractable, because the largest term is where effort pays.

The metric window is the first delay and usually a few tens of seconds. Shrinking it increases noise and, in a proportional controller, produces an oscillation whose period matches the window. For a service with a one-minute metric window and a controller interval of fifteen seconds, three controller passes happen per metric update, which is not harmful but multiplies the number of actuation steps and therefore the number of near-duplicate scale events.

Scheduling and capacity acquisition is the second delay, and its size depends entirely on whether the cluster has spare nodes. On a cluster with headroom, a pod schedules in a few seconds. On a cluster that must add a node, the delay includes the cloud provider’s instance launch, which is measured in tens of seconds for the fastest cases and minutes for the slowest, and includes quota checks that can fail outright. A plan that relies on node autoscaling to serve a fast ramp is relying on the slowest term in the chain.

Image pull is the third and is the one most often underestimated. An image of a few hundred megabytes pulled from a registry at a rate of tens of megabytes per second takes on the order of ten seconds per replica, and a new node pulling the same image for several replicas shares an uplink, so the aggregate time scales with replica count per node. Pre-pulling images onto a node pool, or using a smaller runtime image, removes most of this term; the arithmetic is unglamorous and the effect is large.

Application startup is the fourth. A process that must open a connection pool, warm a cache, or load a model can take substantially longer than the rest of the chain. Warm-up work is not optional if it is on the critical path of becoming ready, and the readiness probe interval adds a final increment: a replica is counted healthy only after the probe has succeeded, so the probe period is a floor on the detection of readiness.

Adding those terms gives a realistic lag of a minute in the best case on a warm cluster, and several minutes when nodes and images are cold. The plan’s headroom must cover load growth over that interval, not over the planning horizon.

Headroom as a function of lag

If demand can grow by a factor $g$ per unit time at its fastest, and the reaction time is $t_r$, then capacity must exceed the load at the moment of the decision by a factor of at least $1 + $ the growth over $t_r$. In terms of a multiplier, if load at time $t$ is $\Lambda(t)$ and the scaling decision uses a measurement from $t - t_m$ and completes at $t + t_a$, the fleet must handle $\Lambda(t + t_a)$ while sized from $\Lambda(t - t_m)$. The required static headroom for a service that relies on reactive scaling is

$$h \geq \frac{\Lambda(t+t_a)}{\Lambda(t-t_m)} - 1$$

which for exponential growth at rate $r$ per second is approximately $e^{r(t_a+t_m)} - 1$. A doubling time of five minutes and a total lag of ninety seconds gives $e^{0.693 \times 1.5} - 1 \approx 1.8$, meaning the fleet must already be at nearly three times the observed demand’s capacity. That figure is a direct statement that reactive scaling cannot follow a fast ramp, and the correct conclusion is architectural: the ramp must be absorbed by pre-provisioned capacity, a load balancer with backlog, or a cheaper degradation path, not by the autoscaler.

import math

def required_headroom(growth_per_second, metric_lag, actuation_lag):
    # Exponential growth assumption; conservative for compound ramps.
    return math.exp(growth_per_second * (metric_lag + actuation_lag)) - 1

def doubling_time_to_rate(doubling_seconds):
    return math.log(2) / doubling_seconds

# A service doubling every ten minutes with a 30 s metric window
# and 120 s actuation needs headroom for 150 s of growth.
print(required_headroom(doubling_time_to_rate(600), 30, 120))

The same function run backwards answers a design question: given a headroom budget, what is the longest lag the service can tolerate? Expressing the plan as a lag budget makes the trade explicit and puts the choice of metric window, image size, and probe interval into the same conversation as the replica count.

Pre-provisioning and the plateau

The alternative to reactive scaling is a schedule: capacity set in advance from a forecast, with reactive scaling as a correction for forecast error rather than as the primary mechanism. The schedule has to be produced from historical load at a granularity fine enough to capture the daily shape, and it has to lead the ramp rather than follow it. Leading means the capacity for the morning ramp is in place before the ramp, which costs the difference between the forecast and the actual for the lead interval.

The plateau argument sets the floor on how much scheduled capacity is worth having. A scheduled baseline sized at the low-traffic level costs little and removes the lag for the smallest variation; a baseline sized at the median costs more and removes it for most days. The choice depends on the distribution of daily peaks: where the peak is predictable to within a few percent, the scheduled baseline can be set near it and the reactive layer never engages; where the peak has a heavy tail, the reactive layer’s lag sets the headroom and the baseline only shortens the loop’s starting point.

The interaction with failure is where scheduling pays twice. A zone failure removes capacity instantly, and no autoscaler replaces it in less than the lag. A scheduled baseline with cross-zone symmetry means the surviving zones are already carrying the load they will need to carry; a baseline sized exactly to each zone’s own load means the loss of a zone triggers a scaling event at the worst moment. The failure arithmetic belongs with the redundancy calculation rather than the load calculation, and the two are compared directly in Capacity Failure Modes Under Peak Load.

Backlog instead of capacity

Every system has a buffer between arrival and service: a socket accept queue, a connection pool queue, a message broker, a load balancer’s pending requests. A buffer absorbs the provisioning lag, converting a capacity shortfall into latency for the duration of the lag. The size of the buffer determines how much lag is survivable, and the cost of the buffer is the latency of requests that wait in it.

This is a genuine trade and its direction is worth stating precisely: a larger backlog extends the load level at which the system can operate without losing requests, at the cost of increasing latency for everything that passes through it during the burst. If the latency objective has room, the buffer is free capacity. If it does not, the buffer only postpones the failure and makes it larger, because requests accumulate and then time out together.

The pathological case is a backlog whose consumers retry. A request that waits, times out, and is retried consumes capacity twice, and the retry load is proportional to the backlog, which makes the backlog self-reinforcing. That positive feedback is the mechanism behind retry storms and is the reason a provisioned buffer must be paired with an admission control decision: when the backlog exceeds a threshold, shedding is cheaper than queueing.