Connection Draining and In-Flight Request Handling at Cutover

A deployment that replaces every instance of a service will terminate connections that were in use at the moment of replacement. If those connections carried requests, the clients see failed requests, and no amount of upstream routing cleverness prevents it: the routing layer can only choose between endpoints, and a terminated endpoint is the origin of the error. Draining (the discipline of removing an endpoint from the pool, waiting for its open work to finish, and only then stopping the process) exists to close that gap.

The mechanism looks simple and has three independent failure modes: taking too long to leave the pool, closing the listener before the pool has stopped using it, and cutting open connections when the process exits.

The deregistration ordering problem

Removal from a load balancer is not instantaneous for the router. In Kubernetes, marking a pod for deletion sets its deletionTimestamp, which removes it from the endpoint list that the dataplane consumes; the kubelet then runs any preStop hook and finally sends SIGTERM to the container’s main process. Those events are asynchronous across three components (the API server, the endpoint controller, and the kubelet on the node) so there is an interval during which the pod is dying but the router has not yet been told. If the application responds to SIGTERM by closing its listener immediately, requests the router is still forwarding to that address get a connection refused or a reset. A short sleep in a preStop hook exists solely to cover that propagation delay; it is a race workaround, and its correct length depends on control-plane latency, not on the application.

The same shape appears outside Kubernetes. A cloud target group deregistration, a service-discovery lease expiry, and a health-check-based removal all take time to become visible to whichever component makes the forwarding decision. The invariant is that the endpoint must remain serveable until the router has provably stopped sending to it, and the only reliable proof is that the router’s view has been updated.

Keep-alive turns one connection into many requests

HTTP/1.1 defaults to persistent connections; RFC 9112 defines the semantics and the Connection: close mechanism for requesting an end to reuse. A client holding a keep-alive connection to an instance will send further requests on it, and each of those arrives without any new routing decision — the load balancer chose that upstream when the connection was established. In HTTP/2 the same is true at stream granularity: one connection carries many concurrent streams, and closing it aborts all streams in flight.

This is why the count of “requests in flight” is not the useful metric for draining; the useful metric is open connections and their idle timers. A graceful shutdown should therefore stop accepting new connections before closing existing ones. Two orderings are available: close the listening socket and let existing connections drain (the process can no longer receive new work, and the router observes a connect failure if it still sends), or keep the listener open and reply to new requests with a refusal that the router interprets as a signal to remove the endpoint. Most servers implement the first and rely on the router having already stopped. The idle timeout matters more than most operators expect: a connection that is idle for 60 seconds and then reused means the upstream must live at least that long after the last request unless the router is willing to retry on a fresh connection.

What a graceful shutdown actually has to do

Three things happen in order, and each has a deadline:

1. Stop accepting new work — close or deregister the listener, and for a job processor, release the queue lease.
2. Let in-flight work complete — track outstanding requests and outstanding background tasks. Most HTTP frameworks expose a shutdown hook that waits for the handler goroutines/threads to return; without it, the process exits with requests half-written.
3. Exit — after all done, or after a hard deadline, whichever is first. The hard deadline must be shorter than the orchestrator’s kill timeout, or the orchestrator will send SIGKILL and defeat the whole exercise.

The last point is the most commonly misconfigured. Termination sequences give a graded escalation: signal, grace period, forced kill. A gracePeriodSeconds of 30 with a shutdown deadline of 25 leaves a five-second margin for the framework to unwind; the reverse ordering (deadline 60, grace 30) guarantees a forced kill for any request that takes longer than 30 seconds, producing an error the operator believed they had engineered away. Requests whose natural duration approaches the grace period (long uploads, long-polling endpoints, streaming responses) cannot be drained at all inside it, and the honest options are to move them off the replaceable fleet, to shorten their duration, or to accept client visible failure and make the client retry.

Timers and where the numbers come from

Draining is a chain of timeouts, and the chain only works if the numbers are ordered. From the outside in:

• Client keep-alive idle timeout. Bounds how long a client will hold a socket it is not using. Commonly tens of seconds.
• Load balancer idle timeout. Must exceed what the application needs, otherwise the router closes connections that the application considers live. An nginx upstream with keepalive_timeout and a backend whose own keep-alive is longer can produce the classic race where the backend answers a request on a socket the proxy had just closed; the proxy’s retry behaviour determines whether the client notices.
• Deregistration delay / grace period. Must exceed the longest in-flight request plus the propagation delay.
• Application shutdown deadline. Must be shorter than the grace period minus a small margin.
• Orchestrator kill timeout. The backstop that ends everything.

Every one of these is an upper bound on request duration somewhere in the path, so the longest legitimate request dictates the whole chain. Where the request is a stream that can legitimately last minutes, the chain cannot be made to fit, and the design must change instead: return a pointer, use a resumable protocol, or let the connection break and have the client reconnect against the new fleet.

Failure modes that survive correct timers

Timers being right does not make draining safe on its own. Three cases escape it.

Retries and non-idempotent requests. A router that retries a failed request on another endpoint can duplicate side effects. If a request had reached the old instance’s handler and mutated state before the connection was cut, and the router retried it elsewhere, the mutation happens twice. Retry logic belongs with idempotency keys or with safe methods only; RFC 9110’s distinction between safe, idempotent, and unsafe methods is the relevant contract.

Connections the router cannot see. If the application opens its own outbound connections (to a database, a message broker, a peer) closing the listener says nothing about those. A shutting-down process that has released its inbound listener but is still holding an exclusive lease, a lock row, or a partition assignment can block its replacement from starting. Background consumers need their own drain path: stop fetching, finish the batch, commit offsets, release the lease.

Load-dependent failures. Draining increases the request rate on the remaining instances in exact proportion to the fraction removed, and a fleet sized for steady state may not tolerate it. If each of ten instances is at 60% utilisation and one leaves, the rest go to roughly 67%; the arithmetic is linear, but the failure modes (queueing, timeouts, retries) are not. Progressive rollout avoids the step change by removing capacity in small increments (the mechanism described in weighted canary routing) and blue-green accepts the step in exchange for a single symmetric switch, as covered in blue-green cutover mechanics. Draining is the piece both of them depend on and neither of them replaces.