Autoscaling Control Loops: Why the Metric Lags the Demand

An autoscaler is a feedback controller. It measures an output, compares it to a target, and adjusts an input. Everything that makes such a controller hard to tune in practice comes from the loop’s delays: the measurement is an average over a past window, the input takes time to become effective, and the output responds to the input only after a further delay. A controller designed as though the loop were instantaneous oscillates; one designed with excessive damping responds so slowly that it fails to track the load.

The loop, written out

The canonical proportional controller computes a desired replica count as the current count multiplied by the ratio of observed metric value to target value. If the current count is ten, the observed utilization is 80 percent, and the target is 50 percent, the desired count is sixteen. The controller then applies limits — a maximum change per step, a maximum and minimum replica count — and writes the result.

The proportional form is chosen because it is stateless and self-correcting: if the observed value equals the target, the ratio is one and the count does not change. It also has no integral term, so a persistent offset between the observed value and the target is not corrected by accumulating error; it is corrected only by the proportional response to the current reading. That is usually fine, because the target is expressed as a ratio of a resource rather than as a setpoint for an absolute quantity.

The important structural property is that the controller acts on the ratio, not on the trend. It does not know whether utilization is rising or falling; it knows only that it is currently above or below target. A rising trend that has just crossed the target produces the same response as a flat value at that level. Trend information must therefore come from elsewhere — a predictive policy, or a faster metric — or the controller will always be reacting to a value that is already a window old.

The delay stack

Four delays sit between a change in demand and a change in the replica count’s effect.

The metric delay is the first. Utilization is computed from a counter sampled at an interval and averaged over a window. A one-minute average with a fifteen-second scrape interval means the value the controller sees at time t describes roughly the interval from t minus seventy-five seconds to t minus fifteen. The controller is steering by the past.

The reaction delay is the second. The controller runs on its own period, which is typically fifteen to thirty seconds. A crossing that happens just after a run waits a full period before being acted on.

The provisioning delay is the third, and it is the largest. Creating a replica involves scheduling it onto a node, pulling its image if the node does not have it cached, starting the process, and waiting for its readiness probe to pass. Image pull alone can take tens of seconds for a large image on a cold node. The readiness probe’s initial delay and period add to that. A replica that the controller requested at time t may not serve traffic until t plus a minute or more.

The startup delay is the fourth and is often conflated with the third: an application that needs to warm a cache or open a connection pool before it can serve at full capacity is not useful the moment its readiness probe passes. Its effective capacity ramps over a further interval that no platform metric captures.

Summing these, a loop tuned for a one-minute reaction time can easily have a three-minute end-to-end lag. Load that doubles in thirty seconds will have doubled and begun to recede before the first replacement replica serves a request. This is the fundamental reason autoscaling is a capacity-planning tool for trends rather than a defence against spikes: no feedback loop can respond faster than its own delay, and the delay is dominated by mechanics that cannot be removed.

Damping and the three ways it goes wrong

Damping exists to prevent the controller from reacting to noise. The standard mechanisms are a tolerance band, a rate limit on changes, and a stabilization window that requires a metric to remain on one side of the target for a duration before acting.

A tolerance band ignores deviations smaller than some fraction of the target. It prevents the controller from chasing normal variance, and it means small sustained deviations are never corrected. The band’s width is a direct trade between churn and accuracy.

A rate limit caps how many replicas can be added or removed per step. It prevents a single bad reading from causing a large swing, and it means a genuine step change is approached gradually. Combined with the provisioning delay, the rate limit sets the maximum rate at which the fleet can grow, which must exceed the maximum rate at which load can grow or the fleet falls behind permanently.

A stabilization window delays scale-down specifically. Because removing replicas is cheap and fast while adding them is slow and expensive, the asymmetry is deliberate: scale up eagerly, scale down cautiously. A scale-down window of several minutes means a transient dip does not trigger a removal that would have to be undone.

# The three failure shapes of a damped loop.
#
# oscillation   : gain too high, delay too long -> count swings around target
#                signature: replica count changes direction every few periods
# undershoot    : stabilization too long -> fleet stays above target after load falls
#                signature: sustained low utilization, no scale-down events
# saturation    : rate limit below load growth rate -> queue grows without bound
#                signature: rising latency with replica count climbing steadily

Each has a different remedy, and applying the wrong one makes the problem worse. Oscillation is fixed by lowering gain or lengthening the window, not by raising the maximum. Undershoot is fixed by shortening the scale-down window or raising the tolerance’s asymmetry. Saturation is fixed by raising the rate limit or by adding capacity ahead of demand, and lowering gain in that case is exactly wrong.

Metrics that lead and metrics that lag

Utilization is a lagging metric by construction, because it is computed from work already done. Queue depth is a leading one: it rises before utilization does, because requests accumulate in the queue before the workers are saturated. A controller that scales on queue depth can react earlier than one that scales on utilization, at the cost of being noisier — queue depth is more volatile and more sensitive to a single slow request.

Concurrency, the number of in-flight requests per replica, is the metric that maps most directly onto the queueing model. By Little’s Law, in-flight requests equal arrival rate times service time, so a target concurrency is a target for the product of load and latency. This makes it sensitive to latency changes in a useful way: if latency rises because a downstream dependency slowed, concurrency rises and the controller adds replicas even though the arrival rate did not change. That is usually the desired response, since the additional replicas restore throughput, but it can also be a runaway: a downstream service that is saturated gets more concurrent requests from a larger fleet and slows further.

The runaway is a genuine feedback hazard and it has no clean solution inside the autoscaler. The standard mitigation is a hard maximum on the fleet size, chosen from the downstream service’s capacity rather than from the application’s own needs. The maximum is the point at which the loop stops being self-correcting and becomes self-reinforcing.

The interaction with node capacity

Pod autoscaling changes the number of pods; something else must change the number of nodes. If the cluster has no room for the new pods, they remain pending, and the controller sees the metric stay high and keeps requesting more, producing a growing set of unschedulable pods. The pod controller’s view of success is the replica count it wrote, not the count that became ready, so the two can diverge silently.

The node-level controller reacts to pending pods by adding nodes, but it has its own delay stack: a machine must be provisioned, registered, and made ready before pods can be scheduled onto it. On a platform where nodes are created as instances, that delay includes the instance launch and the node’s own boot and join sequence. The total lag from a load increase to a serving pod can therefore be several minutes even when every controller behaves correctly.

The mitigation is to keep a buffer of unallocated node capacity — a minimum number of spare nodes, or an overprovisioning deployment of low-priority placeholder pods that reserve space and are displaced when real pods arrive. The buffer converts a multi-minute provisioning delay into a scheduling delay. It costs the price of idle capacity, and the correct size is the capacity needed to absorb the fastest load ramp that must be survived, which is the same quantity the rate limit was sized against. Whether the buffer’s nodes can accept the incoming pods at all is a packing problem rather than a count of free cores.

The requests and limits declared on the pods determine whether that buffer is actually usable, since the scheduler places against requests and the node controller sizes against allocatable capacity. A buffer that looks large in vCPU terms can be unusable if the pods’ requests do not fit its shape.