Decomposing the Failover Time Budget

Recovery time objectives are usually written as a single number, and that number is usually wrong in a specific direction: it is smaller than the sum of the steps that have to happen. The reason is that failover is not one operation. It is a pipeline of four stages, each with its own delay, and the pipeline runs sequentially. Total recovery time is the sum, not the maximum, and the largest single term is rarely the one people expect.

The four stages and where their time goes

Every failover, from a single-node process restart to a continental traffic shift, passes through the same stages.

Detection. Something must observe that the primary is no longer serving. Passive detection waits for user requests to fail; active detection probes the primary on an interval. With an active probe on interval p and a failure threshold of n consecutive misses, the worst-case detection delay is approximately n × p, and the expected delay is roughly (n - 1) × p + p/2 if failures arrive uniformly between probes. A probe every 10 seconds with a threshold of 3 gives up to 30 seconds of detection, expected around 25. Shortening the interval shortens detection but increases probe traffic and the chance of a false positive from a single dropped packet; the threshold exists to convert that trade into a policy.

Decision. Once a failure is observed, something must decide to promote a standby. If the decision is made by a single controller, the delay is the controller’s own detection plus its action time, and the controller is itself a single point of failure. If it is made by a quorum, the delay includes a round trip among the voting members. That round trip is bounded by the inter-region propagation floor, so a quorum spanning continents adds tens of milliseconds per vote round, plus whatever time the members take to agree that the failure is real rather than a network partition affecting only the observer.

Promotion. A standby must be told to accept writes, must complete any recovery work (replaying a log, rebuilding an index, warming a cache), and must be ready to answer. The replay and warm-up term is the one that is invisible in architecture diagrams and dominant in practice: a standby that has been receiving log records but never served queries has cold caches, and the first minutes after promotion are served from a cold state. If the recovery time objective counts only the time until the standby accepts a connection, the objective is met while users experience elevated latency.

Redirection. Traffic must be pointed at the new primary. This stage is the most misunderstood, because it depends on what the client caches. If clients resolve a name through DNS, the effective delay is bounded below by the record’s time to live, and in practice by the behaviour of resolvers that do not honour it. DNS-based failover and its caching bound treats that stage in detail; the summary is that redirection can cost anything from seconds to hours, and the operator controls only part of it.

Written as a budget, the pipeline looks like this:

total = detection + decision + promotion + redirection
      = (n × p)   + (q × RTT) + (replay + warm) + (TTL + resolver slack)

Each term has a different owner and a different cost curve. Detection is bought with probe frequency. Decision is bought with quorum size, at the price of latency. Promotion is bought with continuous warm-standby work, at the price of running a second copy of everything. Redirection is bought with lower TTLs, at the price of more DNS query volume and less caching benefit.

The term that usually dominates

For a database failover inside one region, detection and promotion dominate: the quorum is local, the RTT is sub-millisecond, and the replay is the long pole. For a cross-region failover, redirection frequently dominates, because a DNS TTL of 300 seconds alone contributes up to five minutes, and resolvers that ignore TTLs can extend that to hours. A published recovery objective of 60 seconds is therefore not achievable with a 300-second TTL and a client population that caches. The objective and the mechanism must agree.

The common failure is an objective set by the application team and a mechanism chosen by the platform team, with nobody adding the terms. A useful discipline is to write the budget down as four numbers with owners, and to test the sum rather than each term. A failover exercise that measures “time until the standby accepted a write” answers a different question from “time until a client in another region successfully completed a write against the new primary”, and only the second one is the recovery time the objective describes.

Detection trade-offs in detail

Health checks are a classifier, and like all classifiers they have two error rates. A false negative (missing a real failure) adds detection delay. A false positive (declaring failure when the service is fine) triggers an unnecessary failover, which is usually worse than the delay it avoided: it moves traffic, discards in-flight state, and may cause a split-brain if the old primary is still alive.

Three mechanisms reduce false positives without extending detection much:

• Requiring consecutive failures rather than a single miss. A threshold of 2 or 3 filters isolated packet loss at the cost of one or two extra intervals.
• Probing the dependency, not just the process. A process that accepts connections but cannot reach its database is failing from the user’s perspective. Checks that exercise a real read path catch this; a TCP connect check does not.
• Separating shallow and deep checks. A shallow check (process alive, port open) drives fast local restart decisions; a deep check (query succeeds within a bound) drives slower regional failover. Using the deep check for both makes failover trigger on transient database slowness.

The counter-pressure is that deeper checks are more expensive and more likely to fail for reasons unrelated to the service’s health. A deep check that runs a query every second against a large table becomes a load source in its own right.

Fencing is part of the budget, not after it

Any promotion step assumes the old primary has stopped accepting writes, or that its writes cannot be committed. If it has not, the failover produces two writable primaries and the recovery has to include reconciliation. The mechanism that prevents this is a lease or fencing token, and it has a time cost: the new primary must wait until the old primary’s lease has provably expired before it can safely take over. That wait is not optional and cannot be shortened by optimism — it is bounded by the maximum time the old primary could still believe it holds the lease, which includes clock skew allowances and network delay. Fencing tokens and lease expiry works through the arithmetic.

The consequence for the budget is that decision and promotion are not independent. A quorum decision that promotes immediately, without waiting for lease expiry, is faster and unsafe. A quorum decision that waits for lease expiry is slower and safe. The gap between those two numbers is the price of correctness, and it belongs in the budget explicitly rather than being discovered during the first real failover.

What a rehearsal actually measures

A failover rehearsal that is run by an operator who announces it, drains traffic first, and then triggers the switch measures the mechanism and not the pipeline. The stages that dominate a real event — detection with live traffic, decision under partial network failure, redirection with real resolver behaviour — are exactly the ones the rehearsal removes.

A rehearsal that is closer to the real thing keeps traffic flowing, triggers the failure without warning the on-call rotation, and measures time from the trigger to the first successful user-level write observed from a client outside the failed region. That measurement includes all four stages, and it is the only one that can be compared against the objective. Running it periodically also exposes the terms that drift: TTLs get raised during a caching optimization, probe intervals get lengthened to reduce noise, lease durations get extended to reduce spurious failovers, and the sum quietly exceeds the objective without any single change looking unreasonable.

The budget is therefore a maintenance artifact as much as a design one. Its four terms each have a knob, each knob has an owner, and the objective is only meaningful if the sum of the current settings is checked against it whenever any of them changes. A design that cannot state the four numbers has a recovery time objective that is a hope rather than a bound.