Autoscaling Policies and the Cost of Headroom

Autoscaling is a control loop, and every control loop has a sensor, a setpoint, an actuator, and a delay. The cost of an autoscaling configuration is determined by all four, not just by the setpoint. Two policies that both target seventy percent utilization can differ in cost by a large factor because one reacts in seconds and the other in minutes, or because one measures the right signal and the other measures a proxy that moves after the load has already arrived.

The scaling signal determines what the loop can see

A horizontal policy compares a measured quantity against a target and adjusts the replica count. The measured quantity is usually an average of a per-instance resource metric: CPU utilization across pods, or requests per instance, or a queue depth. The choice of signal sets the loop’s blind spot.

CPU utilization is available everywhere and is a poor proxy for load on any workload that is not CPU-bound. A service that spends most of its time waiting on a database has low CPU at every load level, so the loop never scales, and the cost is paid as latency rather than as instances. A service that is CPU-bound at low concurrency and I/O-bound at high concurrency has a utilization curve that flattens, so the loop stops scaling exactly when the work per request grows.

Queue depth or in-flight request count measures demand directly and reacts before latency degrades. The trade is that the signal is application-specific and must be exposed by the service, so it is not available from an off-the-shelf metric.

A per-instance average also hides distribution. Ten instances at seventy percent average can be seven at a hundred percent and three at zero, or all ten at seventy. The average cannot distinguish the two, so the loop’s decision is the same while the correct decision is different. The maximum across instances is the conservative signal, and it costs more because it scales on the hottest instance rather than the mean.

Headroom is a purchase, and it has a monthly price

The gap between the capacity the loop maintains and the capacity the load requires is headroom, and headroom exists for three reasons: to absorb the delay before the loop reacts, to absorb the granularity of the instance size, and to absorb the failure of one or more instances.

Each of the three has a cost that can be stated per unit. If a policy maintains a target of seventy percent utilization, then thirty percent of the provisioned capacity is headroom at steady state, and its cost is thirty percent of the compute bill. That figure is not a rule of thumb; it follows from the setpoint. Moving the target to eighty-five percent reduces headroom to fifteen percent and reduces the compute line by roughly the same proportion, at the cost of less tolerance for a burst between the load arriving and the loop reacting.

The reaction window is the part that cannot be tuned away. A loop that samples every minute and requires a sustained breach before acting takes at least that long to begin scaling, and then the new instances need to start, pass health checks, and warm their caches. The total delay is the sum of the sampling interval, the confirmation window, the provisioning time, and the warm-up time. Capacity must cover the load at the end of that delay, not the load at the moment the decision was made. A burst that triples load within the delay window must be served by headroom, because no scaling action can complete in time.

This is the reason autoscaling and load patterns have to be considered together. The setpoint itself is not free to choose: as latency targets and the capacity floor they imply shows, the tail of the response-time distribution diverges as the target utilization approaches one, so the objective fixes a ceiling on how low the headroom can go.

A workload whose load rises gradually needs little headroom because the loop can track it. A workload whose load steps from one level to another needs headroom sized to the step, or a schedule, or a queue that absorbs the excess.

Scaling down is slower than scaling up, and that asymmetry is deliberate

Most policies scale out quickly and scale in slowly, requiring a longer period of sustained low utilization before removing capacity. The asymmetry protects against oscillation: a workload hovering at the target would otherwise add and remove instances repeatedly, and each cycle pays a start cost and a warm-up cost.

The cost of the asymmetry is that the fleet retains capacity after a spike has passed. The retention period is a direct cost multiplied by the frequency of spikes. A workload with a spike every hour and a fifteen-minute scale-in delay keeps the peak fleet for a quarter of the time, so its average fleet size sits above its average load by a margin that depends on spike amplitude and interval.

A second asymmetry is that scaling in is not free even when it happens. Removing an instance terminates in-flight work, and a service that must drain connections takes time to do so; a batch worker that must finish its current item holds the instance until the item completes. A policy that ignores drain time will either terminate work or hold capacity longer than intended.

import math

def fleet_required(load_rps, per_instance_rps, target_util, spike_factor=1.0):
    """Instances needed so that capacity covers load at the far end of the reaction window."""
    effective = load_rps * spike_factor          # load once the loop could have reacted
    per_instance = per_instance_rps * target_util
    return max(1, math.ceil(effective / per_instance))

The function makes the two levers explicit: the target utilization and the spike factor applied over the reaction window. Neither can be set without knowing the load’s shape, which is why a policy copied from another workload rarely transfers.

Requests and limits are a scheduling cost, not just a runtime setting

In a scheduler that places pods by declared requests, the request is the quantity the scheduler reserves on a node. A request set well above actual usage causes nodes to be under-utilized while the scheduler reports them as full, and the cost is the difference between the reserved capacity and the used capacity across the whole fleet. A request set below actual usage causes contention and eviction, which costs latency and, in the worst case, repeated restarts.

The aggregate effect is large because it is multiplicative. If every pod requests twice what it uses, the fleet carries twice the capacity it needs, and the autoscaler cannot help because it scales on the pods’ own utilization, which is measured against the inflated request. A pod using half its request reports fifty percent utilization at all times, and a policy targeting seventy percent will never scale out, while a policy targeting forty percent will scale out on an idle service. Utilization relative to request is a self-referential signal, and its meaning depends entirely on whether the request was set honestly.

The correct construction is to derive requests from observed usage at a high percentile and to set limits, where limits are enforced, above the request with enough margin that the workload is not throttled at its normal peak. The gap between request and limit is burst capacity that the scheduler does not reserve, so it is free at the scheduling level and contended at the runtime level.

The interaction with commitments

A commitment purchased against a fleet that autoscales has a utilization problem described in commitment instruments and break-even utilization: the committed quantity must be consumed every hour to earn its discount, and a fleet that scales down at night consumes less than the daytime peak. Sizing the commitment to the nightly floor and letting the autoscaler cover the day is the combination that works, and it requires the commitment to be expressed in the same units as the floor.

The reverse interaction also exists. A commitment large enough to cover the peak removes the marginal cost of running at peak, so the autoscaler’s scale-in behaviour becomes the only remaining lever on spend. A policy that retains capacity for fifteen minutes after a spike, when the capacity is already paid for, wastes nothing in cash terms but does consume the commitment that could have covered other work. The two mechanisms are not independent, and treating them as separate programmes produces a fleet that is simultaneously over-committed and over-provisioned.

Where the loop cannot help

Autoscaling cannot reduce the cost of a workload whose floor is high. If the minimum replica count is set by a latency requirement, a redundancy requirement, or a per-instance throughput ceiling, the floor is a fixed cost and the loop only affects the variable portion above it. A service with a floor of twenty instances and a peak of twenty-two has almost nothing for the loop to optimize.

Nor can it reduce the cost of capacity that is provisioned outside the loop. Databases, caches, and message brokers are typically scaled by different mechanisms with different constraints, and their capacity is often a larger share of the bill than the compute fleet. A cost programme focused on the autoscaled tier and blind to the stateful tier optimizes the smaller half. The stateful tier’s cost is dominated by the same headroom logic but with a much longer reaction window, since resizing a database is not a matter of minutes, and that difference is what makes provisioning decisions for stateful systems effectively fixed commitments.