Autoscaling as a Feedback Loop: Gain, Lag, and Why It Oscillates

A replica count that is fixed in a declaration is a capacity decision made once, at deploy time, by someone who does not know the future load. An autoscaler replaces that decision with a loop: measure a utilisation metric, compare it to a target, and adjust the count. The loop is a controller like any other, reconciling desired state against observed state, but its observed state is a measurement with a delay, its actuator is a request that takes time to have an effect, and its output feeds back into its own input. Those three properties make it a control system, and control systems have failure modes that pure reconciliation does not.

The control law, and why it is proportional

The law used in horizontal autoscaling is a proportional controller expressed as a ratio. If the current count is N, the measured metric is m, and the target is T, the desired count is

desired = ceil( N * m / T )

The derivation is an assumption of linearity: if N replicas produce aggregate metric m, then the per-replica contribution is m/N, and reaching aggregate target N_d * T requires N_d = N * m / T. Every use of this law is a bet that the metric scales linearly with the replica count, that doubling the replicas halves the per-replica load. That bet holds for CPU utilisation on a request-serving workload whose cost per request is fixed. It fails when the workload has a fixed cost independent of load: a service whose per-replica baseline is a garbage collector and a connection pool does not halve its CPU when the replica count doubles, because half of each replica’s usage is constant. The measured metric then falls more slowly than the formula predicts, the loop keeps adding replicas, and the count runs away from the point where the target is met.

The formula is applied to the aggregate of the metric across the pods the controller owns, not to a per-pod average computed independently, and the difference matters when pods are heterogeneous. If the metric is a sum over pods, a pod that is failing to report is simply absent from the sum, so the aggregate is understated and the controller concludes there is spare capacity. The remedy is to require the metric to be present for every pod before the loop acts, and to treat missing metrics as a reason not to change the count rather than as zero.

The ceil is not cosmetic. Without it, a count computed as 7.2 would round to 7, leaving the aggregate below target indefinitely and producing a permanent steady-state error of a fraction of a replica. The ceiling biases the loop upward by less than one replica, which is the cheaper of the two errors.

A deadband suppresses action near the target. If the measured-to-target ratio is within a tolerance band, ten percent in either direction as the common default, the count is left unchanged. The deadband exists because the metric is noisy: a workload whose utilisation fluctuates by a few percent around the target would otherwise cause a scale event on every evaluation, and each scale event is not free.

Lag enters at four points and none of them can be removed

The loop’s stability depends on the delay between a change and the measurement of its effect. A proportional controller with no delay is stable for any gain; the same controller with delay can oscillate at a gain that would otherwise be fine, because it keeps applying a correction based on information that is already stale. In an autoscaler there are four delays in series.

The first is the metric sampling interval. Resource metrics are collected by an agent on each node, aggregated by a metrics service, and served over an API. The aggregation window is typically measured in tens of seconds, so the value the controller reads describes the recent past rather than the present.

The second is the control loop’s own period. The controller evaluates on a fixed interval, so a change in load is observed at the next tick at the earliest.

The third is the actuator’s latency: creating a pod, scheduling it, pulling its image, starting its container, and passing readiness takes time measured in seconds to minutes, and the new replica contributes nothing to capacity until the last of those steps completes. The metric the controller used to compute the desired count was measured before the pod existed; by the time the pod serves, the load may have changed again.

The fourth is the feedback path’s own length: the new pod’s usage appears in the metric only after the next aggregation window, which is the same interval as the first delay. So the loop’s total delay is roughly the sampling window plus the pod startup time, and the controller cannot evaluate the effect of its own action any sooner.

The consequence is a hard constraint on how aggressively the loop may act. A controller that reacts to every sample with full gain will, when load rises, add replicas; the replicas take a minute to serve; during that minute the metric stays high because the load has not been absorbed; the controller reads the still-high metric and adds more. When the first batch finally serves, the metric collapses and the loop removes the excess. The oscillation has a period of about twice the total delay and an amplitude set by how much was added during the blind window.

The two mechanisms that damp it, and what they cost

The first damper is a stabilisation window on scale-down. Rather than acting on the latest recommendation, the controller takes the highest recommendation observed within a recent window, several minutes by default, and scales down only to that. The effect is that a brief dip in load cannot remove capacity that will be needed again shortly. The cost is that a genuine and permanent reduction in load is honoured only after the window has elapsed, so the cluster over-provisions for that period. The asymmetry is deliberate: adding replicas too slowly causes latency, removing them too quickly causes errors, and the second is worse.

The second damper is a rate limit on scaling. The controller will not double the count beyond a factor, nor add more than a fixed number of pods, in one step, and similarly for removal. The limit is a statement that the loop’s model is only valid for small perturbations: the linear assumption behind the proportional law holds near the current operating point and degrades far from it, so a large step is computed from a model that does not apply. The cost is that a sudden tenfold increase in load is met with a sequence of steps, and the workload is under-provisioned for the time it takes to walk up.

Both dampers make the loop slower in the direction where speed is most valuable and preserve the loop’s stability. Neither removes the fundamental delay, so a cluster sized by the loop still needs headroom for the worst-case time to add capacity, which is the sum of the actuator’s steps and one sampling window.

Metrics that make the loop behave, and metrics that break it

The choice of metric determines whether the proportional law has any chance of being right, because the law assumes the metric is a monotone function of the load that the replica count can reduce.

CPU utilisation satisfies this for request-serving workloads with a fixed cost per request, and it is the default for that reason. It fails for workloads that are not CPU-bound: an I/O-bound service whose replicas are waiting on a database has low CPU regardless of load, so the loop sees spare capacity and never scales, while latency grows without bound.

A queue depth is a better signal for a worker pool, because it is directly proportional to backlog and therefore to the number of workers needed. Its problem is the opposite one: it is unbounded above, so a single spike produces an enormous recommendation, which the rate limit then truncates. A per-replica request rate has the cleanest linear relationship and the weakest relationship to actual saturation.

Latency as a scaling signal is the most intuitive and the least stable. Latency is not proportional to replica count in the region that matters: it is flat while the system has headroom and rises steeply past saturation, so the proportional law computes a small correction while the system is fine and then a wildly wrong one at the moment of saturation. Using latency as a target means operating on the knee of a curve, where the derivative is largest and the measurement is noisiest.

The metric must also be attributable to the pods being scaled. A metric aggregated over the whole service, including pods owned by a different controller, makes the loop’s output a function of a quantity it does not control, which is the classic recipe for two controllers fighting over one resource.

Vertical, horizontal, and the interaction

Scaling the count is not the only actuator. Adjusting the resource requests of existing pods, so-called vertical scaling, changes the scheduler’s accounting and requires the pods to be recreated to take effect, so it is slower and more disruptive. Its advantage is that it works for workloads that cannot be replicated: a singleton with a large in-memory index can only be given more memory, not more copies.

The two interact through the scheduler. Increasing a pod’s request may make it unschedulable on its current node, which forces a reschedule and, if the cluster is full, a period of unavailability. The cluster-level autoscaler adds a third loop: it observes pods that cannot be scheduled and adds nodes, which is an actuator with a delay of minutes rather than seconds. The three loops are nested, and the nesting is a stability problem in its own right: a node is added in response to pending pods, the pods schedule, the load spreads, the pending signal disappears, and the node is removed after its own stabilisation window, which may be shorter than the time it takes for the next burst to arrive.

The ordering that avoids most of this is to make the cluster-level loop slower and less sensitive than the pod-level loop, so that node additions respond to a sustained deficit rather than to a transient. Reversing the relationship produces a cluster that adds and removes nodes in phase with the workload’s own fluctuations, paying for the churn in both directions.

The measurement delay at the root of all of it is the same delay that makes a canary analysis require a window longer than the step change, discussed in canary releases and progressive delivery. And the loop’s output is a change to a declared replica count, which is then carried out by the mechanism in rolling updates, surge, and the arithmetic of availability. So an autoscaler’s step is a rollout in miniature, with the same bounds and the same dependence on readiness as the gate.