DNS Failover and the Resolver Caching Bound

Pointing a name at a different region is the cheapest failover mechanism available: it requires no client change, no load-balancer API, and no coordination with anything that holds a connection. It is also the slowest, and the gap between the TTL an operator sets and the time traffic actually moves is the source of most failed recovery-time commitments. The mechanism is worth understanding at the record level, because the delay is not a bug in the resolver software; it is the caching design working as specified.

What a resolver actually holds

When a stub resolver on a client asks its configured recursive resolver for api.example.com, the recursive resolver walks the delegation chain from the root, asks the authoritative server for the zone, and receives an answer with a TTL. It caches that answer for the TTL in seconds, then returns it to the stub, again with a TTL (typically the remaining time). The stub resolver and the client application may then cache the address independently — operating system resolvers do, and many language runtimes and HTTP client libraries keep their own connection pools keyed by hostname, which means they may never re-resolve at all while a connection stays healthy.

So there are at least three caches between the authoritative server and the socket: the recursive resolver, the stub/OS resolver, and the application’s connection pool. A TTL of 60 seconds bounds only the first. RFC 1035 permits resolvers to cap TTLs downward but not upward; a resolver is free to serve a cached record until the TTL expires, and the specification does not require it to re-query early because the operator changed the zone file. Editing a record does not invalidate anything that has already been cached.

Two consequences follow. The worst-case time for a change to reach all clients is the TTL of the record as served before the change, plus the time for the last resolver holding it to expire, plus the time for the client to re-resolve and reconnect. If a zone previously published a 24-hour TTL and the operator lowers it to 60 seconds at the moment of failover, the change has no effect on any resolver that cached the 24-hour value. The correct sequence is to lower the TTL well before the failover — at least one old-TTL period ahead — so that the population of caches has turned over.

The lowering sequence

A workable sequence for a planned switch:

Lower the TTL to the target value, and wait for at least the previous TTL to elapse so every cache that held the old value has expired it.
Perform the switch.
Wait for the new TTL to elapse.
Raise the TTL back for normal operation.

Step 1 is the one that is skipped, and it is the one that determines whether step 2 works. The wait in step 1 is not optional bookkeeping; it is the only way to guarantee that no cache is holding a long-lived copy when the record changes.

For unplanned failover there is no step 1, and the achievable recovery time is whatever the previously published TTL allows. A zone with a 3600-second TTL cannot fail over in under an hour by DNS alone. This is the arithmetic that makes DNS unsuitable as the sole mechanism for a tight objective, and the reason health-checked traffic managers or anycast routing exist as alternatives.

Health checks and the second bound

A DNS-based failover usually involves a health-checked record set: the authoritative server answers with only the addresses of endpoints that pass a check, and withdraws addresses that fail. This improves on static records because it removes the need for a human to edit the zone, but it does not remove the caching bound. A resolver that cached an address before the check failed continues to return it until its TTL expires.

The health check adds its own detection delay, on the same arithmetic as any other probe: interval times failure threshold. A check every 30 seconds requiring 3 failures contributes up to 90 seconds before the record set changes. That delay is in series with the TTL, not in parallel with it, so the pipeline is detection (up to 90 s) then propagation (up to TTL). An objective of 60 seconds cannot be met by this pipeline at any TTL, because detection alone exceeds it.

There is also a subtlety in how resolvers treat the withdrawal. If the record set goes from two addresses to one, a resolver holding the two-address answer keeps returning both until expiry, and clients that pick the first address in the list may keep choosing the dead one. The proportion of clients affected depends on the client’s selection strategy: round-robin, random, or first-address-preferred. First-address preference is common in simple clients and produces the worst outcome, because the withdrawn address remains first in the cached order.

Negative answers are cached too

A failure mode that surprises people is the cached NXDOMAIN or NODATA. RFC 2308 defines negative caching, and a resolver that receives a negative answer caches it for a period derived from the SOA record’s minimum field (or the SOA TTL, depending on the resolver’s interpretation). If a name is briefly misconfigured during a change — a typo in the zone, a delegation removed and re-added — resolvers can hold the negative answer for minutes to hours. The visible symptom is that the name works from some networks and not others long after the zone is correct.

The mitigation is the same as for positive records: keep the SOA minimum low if fast changes are expected, and avoid publishing a name that will be withdrawn and re-added. For failover specifically, prefer changing the address a name points to rather than removing and recreating the name.

What DNS cannot do, and what covers the gap

The bound is structural, so the design question is what else moves traffic.

Connection-level redirection avoids DNS entirely for clients that already hold a connection, but it requires the client to connect to something that can route to multiple regions — a global load balancer with an anycast address, or an address that is itself stable while the backend changes. This is the mechanism that makes sub-minute regional failover possible for HTTP traffic, because the client’s name resolution never has to change. It does not help protocols where the client resolves once and holds a long-lived connection to a specific address, such as many database drivers.

Anycast removes the failover decision from the client side altogether: the same address is announced from multiple locations, and BGP routing sends each client to the nearest announcement. Withdrawing an announcement shifts traffic, and the shift is bounded by BGP convergence rather than DNS TTLs. The cost is that anycast routing is not load-aware — a client’s packets go to the topologically nearest point regardless of that point’s capacity — and that a route change can move a connection’s packets mid-flow, which breaks connection-oriented protocols unless the flow is re-established.

Client-side awareness is the last resort: the application knows about multiple endpoints and retries against a second one. This works when the client is controlled by the same organization, and it moves the recovery time into the client’s retry policy, where it can be reasoned about directly. The failover time budget treats this as the redirection term; the point here is that the term is only small if the client has a second endpoint to try and a policy that tries it quickly.

Choosing a TTL

Given that the TTL is a bound on how long a stale address can persist, the choice is a trade between failover speed and query volume. A 30-second TTL on a name queried 10,000 times per second generates at most 10,000 queries per second to the authoritative servers in steady state — feasible for a large provider but a real cost, and a real load on the authoritative infrastructure. A 300-second TTL cuts that by an order of magnitude and raises the failover bound to five minutes.

The practical compromise for a service with a tight objective is not a very low TTL on the primary name. It is a low TTL only on the name that is expected to change, with the stable names that front it having long TTLs, and a global load-balancing layer in front that absorbs the change. The TTL then bounds only the small fraction of clients that resolve the changing name directly, rather than the entire population.

None of this makes DNS-based failover useless. It makes it a mechanism with a known, calculable latency, which is the right way to treat it. A recovery plan that says “we will update the DNS record” and stops there has not stated a recovery time; it has stated an intention. The number comes from the TTL in force before the change, the health-check detection window, and the client’s own caching and connection behaviour — and the last of those is often the largest and the least controlled.