Capacity Failure Modes Under Peak Load

A capacity plan is judged by what happens at the load it was not sized for. The peaks that matter are not the forecast’s central case but the tail of the forecast: a marketing event, a retry storm, a zone loss that concentrates traffic, or a dependency that slows down and holds resources longer. Each has a characteristic failure mechanism, and the mechanisms differ enough that a plan which covers one does not cover the others. Knowing which mechanism is in play determines whether the right response is more capacity, less work, or a bounded rejection.

Queue overload and the disappearance of headroom

The first mechanism is the one the utilization curve describes. As a queue approaches saturation, waiting time grows as $1/(1-\rho)$, and the response to a small increase in load is a large increase in latency. The operational signature is that throughput is flat while latency rises: the system is doing as much work as it can and the excess is being stored in queues.

The trap in this regime is that the capacity that appears to exist is actually the backlog. When a fleet at 95 percent utilization receives a burst, it can absorb the burst by lengthening queues, so throughput looks stable and the load balancer sees no failures. The cost is paid later, when the queued work either completes very late or times out. A timeout of one second with a service time of ten milliseconds bounds the queue at a hundred requests per slot; beyond that, queued requests cannot complete in time, so the queue is storing work that will be discarded. Under those conditions the fleet’s effective capacity is the rate at which it can serve within the deadline, not the rate at which it can execute, and the difference is the discarded fraction.

The measurement that separates the two is the fraction of completed requests that exceeded their deadline, which is not the same as the fraction that failed. A system that completes ninety-nine percent of requests but whose tenth percentile already exceeds the objective has no capacity for the tail, because the tail is exactly the set of requests that happened to arrive during the busy periods.

When the load peak is caused by the system

The second mechanism is the one that converts a peak into a sustained outage: the system’s response to a peak adds load. The retry and timeout loops described in Feedback Effects That Change Service Demand are the general form, and there are several specific instances worth separating.

Health check removal is the most direct. A replica that is slow but functioning fails its health check, is removed, and the remaining replicas receive its share. If the slow replica was slow because the pool was nearly saturated, its removal makes the others more likely to fail their checks, and the removal cascade can take the whole pool out while every replica is alive. The parameter that determines whether this happens is the health check’s timeout relative to the service-time distribution: a check that fails at a latency the system routinely reaches during peaks is a load-dependent capacity reduction that fires when capacity is scarce.

Failover has the same shape at a coarser granularity. Losing a zone moves its traffic to the others, which raises their utilization by the zone’s share. If the plan sized each zone for its own load at the target utilization, the remaining zones are immediately past the target, and the cascade of timeouts and retries begins. The correct sizing rule follows: each zone must carry the full load at the target utilization, not its own share, and the redundancy requirement is therefore a capacity multiplier rather than an additive count. The arithmetic is the same as the fault-tolerance term in Headroom Arithmetic: Turning a Load Forecast into a Replica Count, applied at the failure-domain level.

Cache warming is the third instance. A cache that is serving at a high hit rate is carrying a large share of the work, and when it is cold, the origin sees the full load. A deploy that restarts the cache fleet, an eviction storm from memory pressure, or a change in key distribution all produce the same event: an instant shift of a multiple of the normal origin load. Sizing the origin for its cached steady state is a plan for the state that exists most of the time and not for the state that exists after a restart.

Cold start and the asymmetry of recovery

Recovery from a peak is not the reverse of the degradation. During degradation the system accumulates state: queues fill, connection pools are exhausted, clients hold retry timers, and caches are evicted in favour of the most recently requested keys. When the trigger is removed, that state drains at a rate set by the system’s own dynamics, not by the trigger.

The clearest case is a pool whose slots are held by requests waiting on a dependency. If every slot is blocked and the dependency recovers, the slots free one by one as the requests complete, and the pool’s throughput returns over a period proportional to the timeout. Nothing the operator does to the dependency shortens that period; only shortening the timeout, or increasing the slot count, does. This is why a capacity plan needs an explicit in-flight capacity figure for dependency failure, computed as the product of the slot count and the inverse of the timeout.

A second form is thundering herd on a warm-up. A fleet that restarts all at once begins with empty caches and cold connection pools, and the first requests through each replica are the most expensive ones. If the restarts are staggered, the warmth of the first replicas serves the traffic while the others warm up; if they are simultaneous, the origin sees a load multiple that it was never sized for. The capacity plan’s statement about rolling restarts is therefore a statement about capacity, not only about availability.

A third form is disk and network queuing after an overload. A storage device that has been driven past its queue depth can remain slow after the offered load falls, because the queue of in-flight operations outlives the requests. The recovery time is bounded by the queue depth times the service time, which is the same arithmetic as the request queue but at a different layer. Measuring the queue depth rather than the utilization is what makes this visible.

Degradation that is worth planning

Not all peaks should be survived with full functionality. The alternative is a documented degradation path, and its capacity argument is that a small amount of shed work buys a large amount of preserved work, because of the nonlinearity of the latency curve.

The cheapest degradation is to stop doing optional work: skip non-essential logging, bypass a recommendation service, serve a stale cached value instead of a fresh one, or reduce the sampling of metrics. Each removes a fraction of service demand, and the fractions add. The capacity freed is then available to the requests that remain, and because the freed capacity moves the system back down the steep part of the curve, a ten percent reduction in demand can produce a much larger reduction in queueing delay.

The second is to reduce the work per request by degrading quality: returning fewer results, truncating a payload, or skipping an expensive aggregation. This changes the service time distribution, which changes the variance term in the queueing model as well as the mean, and the variance effect can be larger than the mean effect when the expensive requests are the long ones.

The third is to shed requests at the edge, as described for admission control. Its capacity property is that the shed requests cost nearly nothing, so the capacity available to admitted requests is the full fleet’s. The danger is that shedding without load shedding at every layer pushes the excess into a retry loop somewhere upstream, where it re-enters as new load. Shedding at the edge is only effective if the client honours the rejection.

def admit(queue_len, max_queue, service_time, deadline):
    # Bound the queue so that the worst-case wait stays under the deadline.
    # Waiting requests ahead of a new one each add one service time on average.
    return queue_len * service_time <= deadline and queue_len < max_queue

The function’s two conditions make different statements. The first is the latency bound, computed from the same reasoning as Little’s Law and adjusted for the service time of the class. The second is the memory bound, since the queue itself is a resource. A system that enforces only the first can allocate unbounded memory to a queue that is within its latency budget, and one that enforces only the second can admit requests that cannot complete in time.

What to record so the next plan is better

The information that makes retrospective analysis possible is not the outage timeline. It is the load, the capacity, and the state at the boundary: the peak rate, the per-replica load distribution, the utilization at the moment the first symptom appeared, the queue depths at each layer, and the fraction of requests that were discarded or timed out. Those five quantities are enough to distinguish queue overload from feedback amplification from cold-start amplification, and each implies a different change to the plan.

Recording them also exposes the operating point that the plan assumed versus the one that occurred. A fleet whose peak utilization was far below its target before the outage has a problem other than capacity, and a fleet whose peak was far above has a forecast problem. That distinction is not visible in uptime numbers, and it is the one that determines whether the next revision of the plan should change the replica count or the forecast.