Blue-Green Cutover Mechanics at the Routing Layer

Two fleets run side by side. One carries production traffic; the other is idle but fully provisioned. A single operation moves the traffic from the first to the second, and if the second misbehaves the same operation is run in reverse. That is the whole idea, and almost all of the engineering difficulty lies in the phrase “a single operation”.

The operation is not a deployment step. It is a change to a routing decision, and the character of the technique (its cost, its reversibility, its failure modes) is determined by which component makes that decision and what it knows about the requests flowing through it.

The switch is a routing table entry, not a deployment

Take an L7 ingress that terminates TLS and forwards to a set of upstream endpoints. A cutover replaces that set: the blue addresses come out, the green addresses go in, and the router begins forwarding newly arrived requests to green. In an nginx-style configuration this is a new generation of the upstream block loaded by a reload signal; the master process validates the new configuration and starts new worker processes, and old workers finish the connections they are already serving before exiting. In a Kubernetes Service the same effect is achieved by changing what the selector matches, which rewrites the endpoint list that the dataplane programs. In a service mesh or managed load balancer it is an API call that updates a target group or a weighted backend.

In every case the change is atomic with respect to the router’s own state and non-atomic with respect to the world. Requests already dispatched to blue complete on blue; the router does not recall them. TCP connections already established keep their upstream binding for the life of the connection, because the router’s forwarding decision was made at connect time — with TLS and HTTP/2 that can be minutes, and with HTTP/1.1 keep-alive it is as long as the client keeps reusing the socket. So a cutover is really “all new connections go to green”, and the tail of old traffic decays at the rate connections close. This is the mechanism behind connection draining, where blue is retired deliberately instead of being cut off instantly.

The practical consequence is that the switch has two halves that must both be correct: the routing change itself, and the removal of the old upstreams. Replacing the endpoint list in one step does both, and does it too fast — the instant blue leaves the list, in-flight connections to blue are either allowed to finish (graceful) or reset (abrupt). Routers differ here, and the difference shows up as a burst of 5xx responses from clients that were mid-request.

Why DNS record changes make poor cutover levers

The tempting version of blue-green is to point the same hostname at green’s addresses and let DNS do the work. It costs nothing to implement and it requires no coordination with the ingress. It also gives up most of the control that the technique is supposed to buy.

A DNS answer is a set of resource records with a time-to-live, and the TTL is advisory. Resolvers are permitted to cache longer than the TTL, some do, and clients are permitted to ignore it entirely. Runtimes that resolve a hostname once at start-up and hold the result for the lifetime of the process (a common default for connection-pooling HTTP clients) will never observe the change at all; those processes keep talking to blue until they restart. Long-lived connections behave the same way regardless of resolver behaviour. Second, when the answer contains several addresses, the client chooses which to use, and clients differ: some take the first record (RFC 1034 ordering is not a preference signal), some take all and round-robin locally, some implement happy-eyeballs-style racing across families. The server cannot express “use green now”.

The result is that a DNS cutover moves a fraction of new traffic per TTL interval, with an unknown and potentially unbounded fraction of long-lived clients pinned to the old addresses. Rolling back has the same latency and the same uncertainty. DNS propagation also cannot be verified positively: there is no way to ask the whole internet whether it has noticed. Ingress-level and load-balancer-level cutovers, by contrast, take effect for the next connection and can be tested with a single request.

What the standby fleet has to be able to carry

Blue-green requires the standby to sustain full production load before the switch is useful, because the point of the switch is to make green the only fleet serving. It also requires blue to remain able to take full load back for as long as rollback is offered. The cost is therefore roughly double capacity for the duration of the window, plus whatever the datastore needs to tolerate both fleets simultaneously. That cost is the price of reversibility, and it is the axis on which blue-green competes with canary and progressive rollout, which pay less capacity and accept slower, statistical confidence instead.

Being provisioned for full load is not the same as being warm for it. A newly started fleet has empty caches, empty connection pools at their configured minimum, no JIT-compiled hot paths, and freshly issued TLS session tickets, so the first seconds after the switch contain the highest handshake rate the fleet will ever see. Autoscaling reacts to observed load, which means it reacts after the switch, not before it. The honest options are to pre-scale green explicitly, to hold the pool sizes and caches warm with an idle load, or to send a small share of traffic first — which is a canary by another name, and the reason the two techniques are usually deployed together rather than in opposition.

State that lives outside the fleet

A cutover is only clean if the two fleets are interchangeable. Anything that binds a client or a piece of work to a specific instance breaks that.

In-memory sessions are the classic case, and the fix is externalisation into a shared store, with the caveat in session affinity across versions that the store’s value shape must also be version-neutral. Queued background work is the subtler case: if both fleets run schedulers, every periodic job runs twice during the overlap window, which for idempotent jobs is harmless and for billing runs is not. The usual remediation is that only one generation holds the scheduler lease.

Caches shared between generations are a correctness hazard, not a performance one. A cache keyed by entity id whose serialised value shape changed between versions will hand the new code an old-shaped value, and the deserialiser is where that failure surfaces. Version-prefixed keys or a flush at cutover avoids it at the cost of a cold cache. Connection pool sizing deserves a line of its own: if each instance opens N connections to a database with a fixed max_connections, then doubling the fleet doubles the demand, and the pool settings that were correct at one fleet size are wrong during the overlap. That ceiling, and the schema constraints that make both versions able to read the same rows, are treated at length in expand and contract migration.

Rehearsing the switch, and the price of going back

Because the switch is one call, it can be tested before it is used. A synthetic request sent to green’s address with the production Host header and, where applicable, the production path (through the same auth proxy, the same WAF rules, the same rate limits) exercises the parts a direct curl to a pod IP does not. The common failure is verifying green only through a path that exists solely for testing: a different ingress route, an internal-only name, or an unauthenticated health endpoint. That path proves the process starts; it does not prove the routing entry, the certificate, or the upstream protocol negotiation.

Reversal is symmetric in mechanism and asymmetrical in cost. Pointing the router back at blue restores the previous code within one connection, but only if everything green wrote is readable by blue. A schema change that dropped a column, a backfill that rewrote rows in a new encoding, or messages published in a new format all survive the code rollback and leave blue reading data it cannot parse — the boundary examined in rollback boundaries. Treat the cutover as reversible, and the cutover plus a destructive data change as a one-way door; schedule the two separately, with time between them during which blue is still viable.