Graceful Shutdown and the Ordering of Connection Draining

Stopping a server is a distributed problem disguised as a local one. The process can close its sockets in any order it likes, but the clients, the load balancer, and the kernel all have their own views of which connections are usable, and none of those views updates instantly. A shutdown that ignores that lag produces connection resets on requests the operator believes were completed before the process stopped.

The sequence that avoids them has four phases, and each phase exists because of a specific way the previous one can leave work in flight.

Stop accepting, and know what that does

Closing the listening descriptor is the first action, because it is the only one that guarantees no new connection can be established. On Linux, pending connections that have completed the handshake but have not been accepted are reset when the listener closes; the client sees a connection reset rather than a refusal, which clients generally treat as retryable but which looks like a server fault in logs. Connections still in the handshake queue are dropped with no packet at all, and the client retries its SYN until it gives up.

The accept loop, if it is blocked, returns an error when the descriptor is closed. Depending on the platform and the descriptor’s sharing, the error may be EBADF or EINVAL, and a loop that treats every accept error as fatal will exit before it has drained anything. The loop should distinguish the shutdown case and stop accepting while continuing to serve the connections it already holds.

This phase separates cleanly from the next one only if the listener is the sole source of new connections. A server that accepts from several descriptors, one per worker with SO_REUSEPORT, must close all of them; closing one leaves the others advertising the same address, and new connections continue to arrive at a process that believes it is shutting down.

Deregister before draining, and wait longer than the check

A load balancer removes an instance from rotation based on its own health checks, and the removal takes effect at the balancer, not at the instance. Even after the instance stops accepting, the balancer may send it requests for one or more check intervals, because the last successful check is still cached as healthy.

The consequence is an ordering constraint that is easy to state and easy to violate: the instance must keep serving for longer than the interval between the moment it signals unreadiness and the moment traffic actually stops. A health check that runs every few seconds with a threshold of several successes means a delay measured in seconds, not milliseconds. During that window the server should still be accepting, because a request that arrives is a request that was routed before the removal took effect and would otherwise be reset.

This is why the usual pattern is a pre-stop delay before the listener closes, rather than an immediate close followed by a wait. The delay is longer than the propagation, and its length is derived from the balancer’s configuration rather than chosen for convenience.

Telling the connections themselves

Stopping new connections does not stop existing ones from sending new requests. A persistent HTTP/1.1 connection is available for another request the instant the current response completes, and the client has no way to know the server is going away.

The signal is in the response: Connection: close tells the client that this connection will not be reused. It has to be added to the response headers before they are written, which means the decision must be visible to the response path rather than made afterwards; a middleware that inspects response headers at the end of the chain is too late, as the ordering discussion in middleware composition explains.

HTTP/2 has an explicit mechanism instead of a header. GOAWAY carries the highest stream identifier the sender will process. Streams above that identifier were never started and can be retried elsewhere immediately; streams at or below it are the ones being finished. RFC 9113 recommends sending two GOAWAY frames, the first with the maximum identifier so that the peer can measure the round trip and know when the second will arrive, and the second with the real cutoff. A server that sends only the final frame risks the client having already sent requests the server refuses.

Because a connection carries many streams, draining is per stream rather than per connection. The server waits for the last stream at or below the cutoff to complete, and closes when the stream table is empty. A single long-lived stream, such as a subscription feed, holds the whole connection open, which is why streaming endpoints need a separate deadline or an application-level signal that the client should reconnect.

The deadline arithmetic

Every drain needs an outer bound, because a client can hold a request open indefinitely and the process must eventually stop. The bound is negotiated with whatever supervises the process: an orchestrator sends a termination signal, waits for a configured grace period, and then sends a signal that cannot be caught.

The grace period must exceed the drain deadline, which must exceed the slowest request the server is willing to complete. If the grace period is shorter than the drain, the forced termination arrives while requests are still running, the kernel closes their sockets, and the client sees a reset after the server had promised to finish. The usual failure is a grace period set to a round number that happens to be shorter than the timeout of some upstream call.

Inside that budget, the drain deadline should be enforced by the server itself, so the shutdown completes with a clean close rather than a kill. A server that waits for handlers to return naturally has no bound at all; a handler blocked on a database call that will not answer until its own timeout expires extends the drain to that timeout, whether or not it is inside the budget.

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
<-ctx.Done()

time.Sleep(5 * time.Second)      // let health checks propagate; keep accepting
srv.SetKeepAlivesEnabled(false)  // Connection: close on subsequent responses
drainCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
if err := srv.Shutdown(drainCtx); err != nil {
    srv.Close() // deadline passed: close remaining connections
}

Closing a connection that has unread input

There is a subtlety in the final close that applies to both protocol versions. If the server closes a socket while unread data sits in its receive buffer, the kernel sends a reset rather than a graceful close, and the peer may lose the response that was already written. This is the reason a server that rejects a request early sometimes takes care to drain a bounded amount of the body before closing.

For a connection being drained normally, the risk is small, because the response is written and flushed before the close. The risk returns when the response is large and the client is slow: a write deadline that fires mid-response produces a truncated body, which for a chunked response is a framing error the client can detect but not repair. Choosing between waiting for a slow client and cutting it off is a policy decision, and the deadline is where it is expressed.

SO_LINGER with a zero timeout changes the close semantics to an immediate reset, which is useful for shedding connections and wrong for draining. Default linger behaviour leaves unsent data in the kernel to be transmitted after the process exits, which is what makes a clean close clean.

Retries at the seam

The client’s behaviour determines whether any of this is visible. A client that reuses a connection which the server closed between requests sees a reset on a request that was never processed, and retrying it is safe regardless of method, because the server never read it. A client that receives GOAWAY retries streams above the cutoff for the same reason.

Requests that were in flight when the drain deadline expired are a different case. They may have been partially processed, so retrying a POST can duplicate an effect unless the operation is idempotent or carries an idempotency key.

The arithmetic that makes shutdown safe, then, is not only about time. It depends on clients distinguishing “never processed” from “possibly processed”, which is a property of the connection state at the moment of the reset and of the identifiers the application chooses. A server that closes connections abruptly erases that distinction and leaves the client no option but to guess.

The last piece in the chain is the component in front of the server, whose own draining has to nest inside the same budget, and whose handling of the headers that describe the original client is the subject of proxy headers and hop-by-hop handling.