Draining a Worker Pool Without Losing or Doubling the Work

Every deployment restarts every worker. In a system with a five-minute lease, a restart handled carelessly stalls each in-flight job for the remainder of its lease, and a rolling deployment that touches every host produces a wave of stalls that lasts as long as the rollout. Handled slightly differently, the same restart produces duplicate executions instead: a worker that stops renewing its claim while its handler keeps running has, from the system’s point of view, died, and its job is handed to somebody else.

The shutdown sequence is the point where the lease design, the retry policy, and the deployment tooling all meet. Each of them imposes a constraint, and the constraints are best satisfied in one order.

What termination actually does

SIGTERM has no special meaning to the kernel beyond its default action, which for SIGTERM is to terminate the process. The grace period is not provided by the operating system; it is provided by whatever supervises the process. In Kubernetes, a container receives SIGTERM and then SIGKILL once terminationGracePeriodSeconds elapses, and that value defaults to 30 seconds. Nothing in the container runtime waits for handlers to finish: if the process has not exited by the deadline, it is killed with no further opportunity to clean up.

The first consequence is that any cleanup which must happen has to be fast enough to fit inside a budget that the supervisor sets, not one the process chooses. A worker that assumes it can take a minute to drain while its pod is allowed thirty seconds will be killed mid-drain on every deploy, and the drain logic will appear to work in testing while failing in the one case it exists for.

The second consequence is that the handler has to be told, rather than discovered afterwards. A process that installs a SIGTERM handler sets a flag and lets the handler observe a cancellation signal; a process that does not install one simply vanishes at the signal, which is the crash case the lease already handles. Both are survivable; only the first is efficient.

The claim that arrives after the stop flag

The shutdown sequence begins with the claiming loop. It must stop issuing claim queries before anything else happens, and it must do so atomically with respect to a claim already in flight — a query that was sent before the flag was set can still return a job afterwards.

The pattern that handles both is a claim loop that checks the flag immediately after a claim returns:

for !stopping() {
    job, err := claim(ctx)
    if err != nil {
        continue
    }
    if stopping() {
        // claimed after the stop was requested: hand it back, do not run it
        release(ctx, job)
        return
    }
    run(ctx, job)
}

Releasing is a conditional update back to a runnable state, guarded by the claim identity, so a job that was somehow already taken over is not disturbed. Without that final check, a worker that has decided to stop still occasionally starts one more job, and its supervisor kills it a few seconds later, leaving the job claimed until its lease expires — the exact stall the drain was meant to avoid.

Releasing rather than running is a choice. A one-second job claimed at the moment of shutdown could simply be executed; releasing it costs a round trip and a different worker’s attention. The decision is worth making on measured handler duration per job type, but the simpler rule — release anything claimed after the stop — is correct for every type and only mildly wasteful.

Finishing, releasing, or waiting for expiry

Jobs already running at the signal fall into three groups, and each has a different correct treatment.

Jobs that will finish within the grace period should be allowed to finish, with their heartbeats continuing. Jobs that cannot finish should be released, not abandoned: a release writes the job back to runnable with its lease cleared, and another worker claims it within a second instead of waiting for the lease to expire. Jobs that hold non-idempotent partial state — a half-written file, a partially applied batch — should neither finish nor be released without first undoing the partial work, or the release hands a second worker a mess.

The deadline that drives the decision has to be derived from the supervisor’s budget, not from a constant. With a thirty-second grace period, cancelling handlers at, say, twenty-five seconds leaves five seconds for the acknowledgement, the release, and process exit. A handler that ignores cancellation past that point does not get to finish; it is killed, and its job stays claimed until expiry. That failure mode is detectable: the count of jobs reclaimed by expiry rather than by release, per deploy, measures how well the drain is working.

Releasing has a cost that is easy to miss. The job’s attempt counter, incremented at claim, has already advanced. A deploy that releases a hundred long jobs every hour consumes an attempt from each of them, and a job with a small attempt budget can be pushed toward its terminal state by deployments alone. Either the counter distinguishes infrastructure interruptions from handler failures, or the budgets are generous enough to absorb a deploy or two.

Heartbeats have to outlive the handlers

The heartbeat described in Visibility Timeouts and Lease Heartbeats exists to prove liveness to the queue. During a drain, the process is still alive and still working, so the heartbeat must keep running until the last handler returns — stopping it at the signal would make the lease expire while the handler is mid-flight, and the job would be claimed by another worker within the lease period.

The most common implementation error here is a heartbeat goroutine tied to the same context that is cancelled on shutdown. The handler’s work context and the lease-renewal context are different lifetimes and need different cancellation: the renewal stops only after the handlers have stopped, and the handlers stop at a deadline earlier than that. Reversing the order — cancelling the renewal with the work — produces duplicate execution precisely during deploys, when concurrency is highest.

Once the handlers have stopped, the remaining jobs are acknowledged or released, and then the process exits. Any claim still held at exit is released either explicitly or by expiry; explicit is better, because it is immediate, but a failed release is not a correctness problem, only a latency one.

The same reasoning applies to components other than the worker. A scheduler that holds a leader lease should release it on shutdown so the next leader takes over on the next tick rather than after the lease’s full duration; a mover that promotes delayed jobs should finish its current batch and stop, leaving any unclaimed rows for its successor. Every component that holds a lease has the same obligation, and a drain that covers only the workers leaves the quietest stalls in place.

Two versions on one queue

A rolling deployment means that for a period, workers running the old code and workers running the new code consume the same queue. Whatever the drain does, this overlap exists, and it constrains what a deploy may change.

A payload written by the new code and read by the old code is the dangerous direction. If the new version adds a field, an old worker ignores it; if the new version changes the meaning of an existing field, an old worker interprets it wrongly and produces a plausible but incorrect result. Renaming a job kind is worse: an old worker that does not recognise the kind either fails it or leaves it claimed until expiry, and the job’s attempts are consumed while the rollout proceeds.

The workable discipline is that payloads carry a version, and a worker refuses to process a version it does not understand rather than guessing — refusing means releasing with a short delay, so the job is picked up by a new worker moments later instead of being failed. Job kinds are added before they are produced and removed only after the producing code is gone, in the two-deploy order that any schema migration follows.

Rollouts also interact with retry and dead-letter states. A batch of jobs promoted to a terminal state during the deploy window will not be retried by the new code, so a deploy that fixes a handler does not by itself recover the failures the old handler produced. Re-drive is a separate decision, made with the error class recorded at promotion, as described in Retry Ladders and Dead-Letter Promotion.

The crash-only counterargument

There is a respectable position that none of this is necessary. A job system already survives a worker that is killed without warning — that is what leases are for — so a deployment could simply kill workers and let the recovery path do its job. Candea and Fox’s crash-only software argument makes the general case: a system that only ever has one failure mode is easier to make correct than one with a graceful path and a crash path that must both be exercised.

The cost of that position is measurable and falls entirely on recovery latency and duplicate execution. A kill leaves every in-flight job claimed until its lease expires, so the stall is the lease duration rather than the drain budget; with a five-minute lease and a deploy every twenty minutes, a substantial fraction of the cluster is permanently in the stall window. It also guarantees that handlers are killed at arbitrary points, which is fine for idempotent handlers and expensive for long ones that must restart from the beginning.

A hybrid keeps the crash path as the only correctness mechanism and uses the drain purely for efficiency: the lease still expires if anything goes wrong, the release is only a shortcut, and the handler’s cancellation is a courtesy. Under that framing the drain can be partial without being wrong, which is the property that makes it safe to ship — a bug in the shutdown sequence degrades to the crash behaviour the system already tolerates, rather than to a lost job.