Feedback Effects That Change Service Demand

Every capacity model in this cluster assumes the load is given: requests arrive at a rate, the system serves them, and capacity is sized to the rate. That assumption fails in overload, and it fails in a specific direction. The components that exist to make a system resilient, which are retries, timeouts, and failover, all respond to failure by adding load. When the failure is caused by load, the response is a multiplier on the cause, and the system’s capacity becomes a function of its own state rather than of the arriving traffic.

The retry multiplier

A client that retries a failed request $k$ times with some probability turns one request into an expected number greater than one. If the retry probability is $r$ and retries are independent, the expected attempts per logical request is a geometric series with sum $1/(1-r)$: $r = 0.1$ gives 1.11 attempts, $r = 0.5$ gives 2, and $r = 0.9$ gives 10. The series is a trap because $r$ is not a constant: it rises as the system gets slower, since more requests exceed the timeout, and the extra attempts make the system slower still.

Layered retries compound rather than add. If a client library retries three times against service A, and service A retries three times against service B, a single logical request that fails at every layer produces up to nine attempts at B, and the amplification is the product of the layer multipliers, not their sum. An architecture with four layers each configured for three attempts can produce eighty-one downstream attempts from one client action. The configuration is reasonable at each layer and unreasonable in composition, and the composition is invisible unless the retry budgets are considered together.

Retries also arrive without backoff or jitter in the common case, which converts what was a uniform stream of failures into a synchronized one. A thousand clients that all retry after the same fixed interval produce a burst a thousand times the normal rate at that instant, which fails the dependency again and lengthens the next interval. Jitter over a window decorrelates the retries, but the mean extra load remains, and the mean is what the capacity plan must accommodate.

The arithmetic that matters for planning is not the retry policy but the effective amplification at the operating point. A service provisioned for a 1.1 multiplier is provisioned for a 10 percent error, and the plan needs a statement of the load at which retries begin to fire, since beyond that point the fleet is serving attempts rather than requests and its effective throughput per unit of user demand drops.

Timeouts as capacity consumers

A timeout is usually discussed as a latency control, and it is also a capacity control, in both directions. A shorter timeout frees a resource sooner: a request that would have consumed a thread for thirty seconds consumes it for two, which raises concurrency capacity by a factor of fifteen for that class of request. A shorter timeout also converts slow successes into failures, which triggers retries, which consume capacity. The optimum is not at either extreme, and it is not the same for every class of request.

The failure mode that makes this acute is the one where the timeout is long relative to the work. A handler blocked on a dependency that has stopped responding holds whatever resource the handler holds, whether a thread, a connection, or a request slot, for the full timeout. If the dependency is down, every in-flight request occupies a slot for the timeout duration, so a pool of $m$ slots serves $m/t$ requests per second where $t$ is the timeout, regardless of how fast the dependency would otherwise respond. Recovery is then gated by the timeout, not by the dependency, and the shape of the recovery curve is a ramp with slope set by $m/t$.

This is the mechanism behind a class of outage in which a dependency recovers but the service does not. The dependency is healthy and idle; the caller is queueing because its slots were all held by requests that have not yet timed out. The capacity plan must therefore budget for in-flight capacity during dependency failure separately from steady-state capacity, and the two are related by the ratio of the failure timeout to the normal service time.

Where the loop turns positive

A negative feedback loop is stabilizing: more load produces more latency, which slows the arrival rate of closed-loop clients, which reduces load. A positive loop is the opposite, and they arise when the system’s response to load increases the load.

The clearest instance is the retry loop described above, but there are several that are less obvious. Load balancer health checks are one: a replica that is slow, not dead, fails a health check under load, is removed from rotation, and shifts its share to the remaining replicas, which makes them slower and more likely to fail theirs. The loop removes capacity in response to a symptom of insufficient capacity. The parameter that governs it is the health check timeout relative to the service time distribution, and a check whose timeout is below the p99 service time is a load-dependent removal mechanism.

Circuit breakers are another, in their half-open state. A breaker that admits a single probe request per interval is conservative when the dependency is down and too conservative when the dependency is up: at a probe rate of one per second per caller, a fleet of a thousand callers generates a thousand probes per second, which may be more than the dependency can take after an outage, and the probes are all short-lived failures. Tuning the probe rate against the fleet size is part of the capacity budget, not a detail of the library.

Queue-based decoupling has its own version. A queue absorbs bursts, which is what it is for, but a queue with an unbounded depth converts a capacity shortfall into unbounded latency, and work that is very old when it is finally processed is often wasted. If the consumers are scaled from the queue depth, a growing queue produces a growing fleet, which is a negative feedback that works, provided the consumers are the bottleneck. If the queue’s growth is caused by a downstream dependency, the extra consumers add load to the dependency, which is a positive loop with a scaling policy as its amplifier.

Admission control as the capacity boundary

The alternative to letting the loop run is to bound the work admitted. Admission control refuses requests at the edge when the system is full, which converts a load-induced failure into a bounded rejection. The capacity argument for it is that a rejected request costs almost nothing while an accepted request that will fail costs its full service time plus the retry it induces. Shedding early is therefore cheaper in exactly the regime where capacity is scarce.

The design question is what to measure for the admission decision. The quantity that predicts queue growth is the excess of arrival rate over service rate, which is not directly observable, but its integral is: the queue length. Admitting while the queue is below a threshold and shedding above it holds the queue bounded, and the bound translates into a latency bound for admitted work. Because the queue length is what is being controlled, the threshold is the parameter, and it is set from the latency objective divided by the mean service time.

Admission control also has a fairness dimension that a capacity plan cannot ignore. Shedding uniformly across all request classes means a cheap, frequent request and an expensive, rare one are equally likely to fail, which is not the intent. Shedding by class, or with priority weights, preserves the capacity for work that matters, at the cost of a policy that has to be maintained as the workload evolves. The Load Balancing Policy and Effective Capacity article treats the related question of which replica receives the admitted work.

Reading a load test that includes the loop

The practical consequence for measurement is that a load test without the retry, timeout, and circuit breaker configuration of production measures a different system. A test that closes the loop at a fixed request rate, with no client retries, produces a clean latency curve up to saturation. The same system in production, with clients that retry, produces a curve that turns upward earlier and does not return along the same path when load is removed.

Running the test at decreasing load after saturation is the diagnostic that separates the two. A system whose latency falls back along the same curve has a stable operating point and a capacity that can be read off the up-going curve. A system whose latency remains high after load is removed has hysteresis, which means some state, usually a queue or a set of held resources, is not draining at the rate the arrivals permit. Measuring the time to return to baseline is measuring the loop’s time constant, and that constant is a property of the timeouts and queue bounds rather than of the hardware. The metastable-failure literature names this state precisely: the system is in a region where the only escape is to reduce load below a threshold well beneath the one that caused the failure.

What the plan must state

A capacity plan for a system with these loops has to record more than a replica count. It must state the retry amplification at the expected error rate, the in-flight capacity consumed during a dependency timeout, the load-dependent removal thresholds for health checks, and the queue bound that admission control enforces. Each of these is a parameter with a units-bearing meaning, and each bounds the load at which the system’s behaviour departs from the model in Variance and the Effective Capacity of a Replica Pool. Stating them in units rather than as configuration values also makes them comparable: a retry amplification of three and an in-flight multiplier of five say that the fleet effectively serves a third of the requests it is sized for while a dependency is timing out.

The numbers are usually less precise than the steady-state analysis, and that is acceptable. What is not acceptable is a plan that assumes the feedback away, because the feedback is what determines whether the capacity that was bought is the capacity that is available when demand is highest. The margin bought for forecast error is a different margin from the one the loops consume, and the two are only additive if the loops stay bounded, which is what admission control exists to guarantee.