Sizing Regional Capacity for the Loss of a Region

A multi-region deployment that is meant to survive the loss of a region needs enough capacity elsewhere to absorb the load. The arithmetic looks trivial: if there are two regions and each normally carries half the traffic, each needs to be able to carry all of it. In practice the required headroom is larger than that, and the shortfall comes from three sources that a simple load-fraction calculation ignores: peak correlation, the nonlinearity of queueing near saturation, and the extra work that failover itself generates.

The arithmetic of N+1 and why it understates

For n regions with equal traffic share 1/n and a requirement to survive one region’s loss, each surviving region must carry 1/(n-1) of the total. For n = 2, that is 100 percent — each region must be provisioned for the full load, which is a factor of two in cost for a factor of one in availability. For n = 3, each must carry 50 percent, so provision at 50 percent of total capacity in each of three regions: 150 percent of the steady-state requirement instead of 200 percent.

That is the standard N+1 calculation, and it assumes traffic is evenly spread and that the remaining regions can take the redistributed load instantly. Neither holds.

Traffic is not evenly spread. A region serving a dense user population carries more than its share, and the region that fails is not chosen to be the smallest. The surviving regions must absorb the failed region’s actual load, not its average, so the sizing basis is the largest region’s peak, not the mean.

Peaks correlate. Regions in the same hemisphere tend to share business hours, and consumer traffic peaks in the evening across a continent. If two regions peak at the same hour, the failover happens at the worst possible moment: the surviving region is at its own peak and must also absorb the failed region’s peak. Sizing for the sum of two peaks is a different number from sizing for the average, and it is the number that matters.

Failover generates load. Redirected clients retry. Retries multiply the request rate at exactly the moment the surviving region is absorbing a new population. Connections are re-established, TLS handshakes run again, caches are cold, and connection pools are empty. A cold cache can raise backend load by a large factor for the duration of the warm-up, because the miss rate starts at one and decays as the cache fills.

The combined effect is that a region provisioned for “50 percent of total” is often not able to absorb the other region’s share at peak with retries, even though the fractions add up on paper.

Queueing makes the last few percent expensive

The reason headroom cannot be trimmed to the exact load fraction is that latency is not linear in utilization. For a simple queueing system, the expected time in system grows with utilization ρ roughly as 1/(1 - ρ). At 50 percent utilization the factor is 2; at 80 percent it is 5; at 90 percent it is 10; at 95 percent it is 20.

The absolute numbers depend on the service time and on the arrival process, but the shape does not: the cost of the last few percent of utilization is disproportionate. A region running at 70 percent steady state has a modest queueing multiplier; after absorbing a failed region’s load it may be at 95 percent, where the multiplier is an order of magnitude worse. Requests time out, clients retry, and the retries raise the arrival rate further, which raises utilization, which raises latency. That is the cascading failure loop, and it is the mechanism by which a capacity shortfall becomes an outage rather than a slowdown.

The practical consequence is that the headroom target is expressed in utilization, not in absolute capacity. A region should be able to absorb the failover load and remain below the utilization at which the queueing multiplier becomes unacceptable — commonly somewhere in the 60 to 70 percent range for latency-sensitive services, which means the provisioned capacity is meaningfully above the failover load itself.

required_capacity = failover_load / target_utilization

failover_load   = own_peak + failed_region_peak + retry_amplification
target_util     = the utilization at which latency stays inside the SLO

With own_peak and failed_region_peak each at 0.5 of total, failover_load is 1.0 of total before retries, and at a target of 0.7 the required capacity is about 1.43 of total per region — not the 1.0 that the N+1 arithmetic suggested. Retry amplification during the transition raises it further.

Where the failover load actually comes from

The load that arrives at a surviving region is not simply the failed region’s traffic. Four components add.

Redirected steady traffic. The population the failed region served, now served elsewhere. This is the component the N+1 calculation counts.

Retry amplification. Clients that had requests in flight when the failure occurred retry. If the client retries three times with no backoff and no jitter, the surviving region sees three times the request rate for those requests until they succeed. Client-side retry budgets — a cap on retries as a fraction of successful requests — are the control that keeps this bounded, and they must be enforced at the client, because the server cannot distinguish a retry from a new request without an idempotency key.

Cache and connection warm-up. A surviving region’s caches hold the data for its own users. The failed region’s users request data that is not in those caches, so the miss rate jumps. If the cache is shared across regions, it may be cold for everyone; if it is per-region, it is cold for the new population. Connection pools to the database must grow to serve the additional concurrency, and each new connection costs a handshake and a backend process slot.

Background work. Batch jobs, index rebuilds, and compaction that were running in the failed region either stop or are rescheduled. If they are rescheduled into the surviving region, they compete with user traffic for the same resources. A design that does not separate batch and interactive capacity will see the batch work starve the interactive path precisely during the incident.

Elastic capacity is not a substitute for provisioned capacity

The instinct is to rely on autoscaling: provision for steady state and let the surviving region scale up when the failure happens. This works only if scaling is fast enough relative to the failure dynamics and if the capacity is actually available.

Three constraints bound it. Instance launch takes time — tens of seconds to minutes, depending on the image and the initialization work. The control plane that performs the scaling is itself a shared dependency, and it is handling a large event at the moment it is needed. And the provider’s regional capacity is finite: if several customers are failing over into the same region at once, the ability to launch new instances in that region is not guaranteed. Capacity that must exist within a bounded recovery time has to be reserved rather than requested, which means paying for it continuously.

The compromise that most designs land on is a floor of provisioned capacity sized for the failover load at target utilization, with autoscaling above it for organic growth. The floor is the N+1 commitment; the autoscaling is for the ordinary case. A design whose floor is sized for steady state has an N+1 plan that depends on a mechanism with unbounded latency.

Testing the number instead of believing it

A capacity plan is a hypothesis about behaviour under a condition that rarely occurs. The only way to know whether the hypothesis holds is to create the condition, or a close approximation, and measure.

Three tests are worth running, in increasing cost:

Load test to the failover load. Drive the surviving region to the computed failover load and observe latency at the target utilization. This validates the queueing assumption and the per-request cost, and it usually reveals that some component saturates earlier than the aggregate number suggests — a connection pool, a DNS resolver, a NAT gateway’s port allocation, or a per-tenant rate limit.

Failover with production-like traffic. Shift traffic as a real failover would, with clients retrying as they actually do, and measure the time to steady state and the error rate during the transition. This is where retry amplification and cache warm-up show up. It also tests the parts of the pipeline that are not capacity: the failover time budget terms are measurable here, and a test that only shifts a synthetic load generator misses the client behaviour that dominates.

Failure during peak. Run the failover at the hour when the surviving region is at its own peak. This is the case the plan is supposed to cover and the case that is least often tested, because it is inconvenient. A plan that only works at 3 a.m. is not a plan.

The measurement to record is not the peak throughput achieved but the latency distribution at the target load, because the SLO is a latency statement. A region that absorbs the failover load at double the normal latency has absorbed it in a way that violates the objective, and the capacity number should be revised upward until the distribution holds.

The coupling with placement decisions

Capacity interacts with the topology choices in the rest of this cluster, and the interactions are not always in the expected direction. Quorum placement that puts two replicas in one region means the surviving region’s database can form a quorum alone after the other region is lost — good for availability, but it means the surviving region’s storage must hold the full data set and serve the full write rate, not half of it. A symmetric three-region quorum with replicas spread evenly cannot form a majority after losing one region, so the failover requires either a reconfiguration or a fourth replica.

Cell-based isolation changes the capacity arithmetic too: cells are provisioned per cell, so losing a region means the surviving cells must absorb the tenants of the failed region’s cells, and the tenant-to-cell map must be reassignable quickly enough that the redistribution happens before clients time out. The reassignment is a routing operation with its own latency, and it belongs in the recovery time budget alongside DNS.

The unifying point is that capacity, placement, and routing are one problem. A number computed for one of them in isolation will be wrong when the other two are chosen, and the error is always in the direction of less headroom than the design assumed.