Retry Budgets and the Load Multiplication They Prevent

A dependency starts failing one request in five. Every caller retries once. The dependency now receives 1.2 times its original load, and it was already at eighty percent utilization, so it fails more. The failure fraction rises, the retry volume rises with it, and the system settles into a state where most of the traffic consists of requests that have already failed once. Nothing in that description is a bug in any single client. Each client implemented a reasonable policy: retry a failed request once, with exponential backoff, applied to a failure that looked transient.

The arithmetic is the problem. Retry amplification is a product of failure rate and attempt count, and the product is invisible when it is evaluated one client at a time.

The multiplier, computed

If a fraction f of requests fail and each failing request is attempted up to n times, the expected load on the dependency is 1 + f + f² + … + f^(n-1) times the original. At f = 0.1 and n = 3 that is 1.11. At f = 0.5 and n = 3 it is 1.75. At f = 0.9 and n = 5 it is 4.10.

The series converges only because the failure probability is treated as constant. In a real dependency it is not: the extra load raises utilization, which raises latency and failure probability, which raises load. A retry policy that is stable at f = 0.1 can be unstable at f = 0.5, and the transition between the two is fast because the dependency’s capacity does not change while the demand against it does.

The distinction that matters is between a dependency that is broken and one that is overloaded. Retrying a broken dependency is cheap and often correct: the request will fail immediately regardless. Retrying an overloaded dependency adds load to the exact resource that is already the bottleneck, which is why a single retry layer can turn a degradation into a saturation event. The system cannot tell the two apart from a single failed response, so the policy has to be safe under the worse assumption.

Which failures are worth another attempt

A retry is only meaningful if the second attempt has a different outcome distribution from the first. Three properties decide that.

The operation must be idempotent, or the effect must be made so by a key. RFC 9110 defines GET, HEAD, OPTIONS, and TRACE as safe, and adds PUT and DELETE to the idempotent set; POST is neither, so a retried POST can create two resources unless the server deduplicates on a client-supplied key. The key has to be generated before the first attempt and reused across all of them, which means it belongs to the request context rather than to the attempt.

The error must be one that can differ. A connection reset before any response bytes arrive is ambiguous: the server may have processed the request. A timeout on a request whose processing is deterministic and slow will time out again. A 503 with a Retry-After header is an explicit instruction, and honoring it is different from retrying immediately. A validation error or a 404 will reproduce exactly.

The attempt must fit in the remaining deadline. An attempt started with less time left than the dependency’s own p99 latency has a low probability of succeeding, and its failure consumes budget that the next attempt could have used. Retry counts are the wrong unit here; the deadline described in Deadline Propagation and the Shared Request Time Budget is the right one, and the retry loop should stop when the remaining time cannot cover another attempt.

Jitter, and why synchronised retries are worse than none

Backoff without jitter synchronises. A thousand clients that fail at the same instant and wait the same 100 ms retry at the same instant, producing a spike that is no gentler than the original failure and arrives when the dependency has had no time to recover. Exponential growth of the delay reduces the frequency of these collisions but does not remove them, because the delays remain deterministic functions of the attempt number.

The standard remedy randomises the delay. With full jitter, the wait is drawn uniformly from [0, min(cap, base * 2^attempt)]; with equal jitter, from [d/2, d] where d is the exponential value. Full jitter spreads arrivals over the whole window and is the variant that minimises collision probability, at the cost of sometimes retrying almost immediately. Decorrelated jitter takes the previous delay as input, sleep = min(cap, random(base, prev * 3)), which avoids the clustering that repeated doubling produces when many clients share a base.

// full jitter: draw uniformly from [0, capped exponential delay]
func backoff(attempt int, base, cap time.Duration) time.Duration {
	d := base << attempt // overflow is bounded by cap below
	if d <= 0 || d > cap {
		d = cap
	}
	return time.Duration(rand.Int63n(int64(d) + 1))
}

Jitter fixes the phase alignment of retries. It does nothing about their volume, because every client still retries independently and the total is still the sum of individual policies.

Denominating the budget in successes

A retry budget replaces the per-request attempt count with a shared allowance. The common form is a token bucket: each successful request deposits a fraction of a token, and each retry withdraws one. When the bucket is empty, retries stop, regardless of how many requests are failing.

This inverts the control. Under healthy conditions the bucket refills continuously and the retry allowance is effectively unlimited at any rate the successes can fund. Under a broad failure, successes become rare, the bucket drains, and retries cease within one refill period. The system degrades to no-retry behaviour, which is exactly the correct response to a dependency that is failing broadly: the retries were not going to succeed and were adding load.

The ratio between deposits and withdrawals sets the maximum amplification. A budget that funds one retry per ten successes caps retry traffic at ten percent of successful traffic, so the dependency can never receive more than 1.1 times its baseline load from this client. Finagle’s retry budget uses this shape, tracking successes and failures over a rolling window and permitting retries only while the observed retry rate stays under a configured fraction of requests.

The failure mode of a naive budget is that a client with very low traffic never accumulates enough tokens to retry anything, so a single-request workload loses retries entirely. A minimum retry allowance per unit time, independent of success volume, restores a small floor without giving up the cap on amplification. The floor should be small enough that it cannot matter when multiplied across many clients.

When the dependency pushes back

Budgets bound what a client inflicts; they do not tell the client when to stop calling. Two server-side signals do.

Retry-After on a 503 or 429 is an explicit interval, and treating it as advisory rather than mandatory is a common way to defeat a server’s own load shedding. A server that sheds load and is immediately re-asked by every client has shed nothing.

Longer delays for repeated failures at the same target are the other signal. A caller that has failed against one instance three times in a row is not learning anything from a fourth attempt, and shifting to a different instance or dropping the target entirely is more useful than shortening the backoff. The mechanisms for that decision belong to Circuit Breaker State Machines and Half-Open Probing, which is the natural place to stop retrying a target that keeps failing.

Retries are not free at the tail

Retry cost is usually discussed in aggregate load, but the latency effect is separate and often larger. A request that fails after one second and succeeds on the second attempt has a two-second latency. If one percent of requests retry once, the p99 of the combined distribution is now drawn from the retried population, and the retry has moved a one-second p99 to something closer to two seconds for the slowest requests. The tail at scale argument applies directly: the requests that retry are disproportionately the slow ones, so retries concentrate in the tail they were meant to repair.

Hedging, where a second attempt is issued in parallel rather than after failure, has different arithmetic and a different cost profile; Hedged Requests and the Arithmetic of Tail Latency works through when the extra load is worth the tail reduction.

The rule that survives all of this is that retry policy is a global resource question. A per-client count answers “how many times should this request be tried”; the question that determines whether the system survives a partial failure is “how much extra load is this client permitted to generate”, and only a budget answers that.