Cancellation Propagation and the Work That Outlives Its Caller

A client disconnects at 300 milliseconds. The server’s handler is inside a database transaction that takes 400 milliseconds to commit. The cancellation signal arrives, the handler’s context is done, and the transaction commits anyway, because the commit was already in flight and the database driver had no way to abort it. The response goes nowhere. The row is written.

That sequence is not a bug in the cancellation mechanism. Cancellation is a request to stop, delivered to code that must cooperate. Everything that does not cooperate runs to completion, and the set of things that do not cooperate is larger than most handlers assume.

What a cancellation signal can and cannot do

Every mainstream runtime implements cancellation as a cooperative protocol. Go’s context.Context exposes a Done channel and an Err method; a goroutine observes cancellation only by selecting on Done or by calling a function that does. Java’s interruption sets a flag on the thread, and blocking methods like Object.wait, Thread.sleep, and the interruptible java.nio channels throw InterruptedException when they see it, while CPU-bound loops and most I/O do not. POSIX signals are asynchronous but only take effect at defined points, and a thread blocked in an uninterruptible syscall does not return until the syscall does.

The common property is that cancellation is checked at boundaries the callee chooses. Code that never checks runs to completion regardless of what the caller wants. Code that checks infrequently runs for up to one check interval after cancellation. Code blocked in a syscall that is not interruptible runs for the duration of that syscall, which for a large write or a lock acquisition can be arbitrarily long.

The practical consequence for a request handler is that cancellation bounds the work only up to the granularity of the checks it performs. A handler that checks between steps has a bounded overshoot equal to the cost of one step. A handler that checks once at the start has no bound at all.

The five places cancellation is not observed

Database drivers. A query sent over a connection is in the server’s hands. Some drivers support a cancellation protocol, either a separate connection that sends a cancel request or a wire-level signal, and the effect depends on the server and on whether the query is in a state that can be interrupted. A commit in progress generally cannot. The gap between “the client cancelled” and “the transaction ended” is the commit duration, and the transaction is durable at the end of it.

Filesystem writes. A write() to a local file returns when the data is in the page cache, not when it is on stable storage. Cancelling after the write() returns but before fsync() completes leaves a window in which the data may or may not survive a crash, and the cancellation has no effect on the fsync(). The syscall is not interruptible in the sense that matters.

Queued messages. A message published to a broker is durable when the broker acknowledges it. If the publish is in flight when cancellation arrives, the message may still be appended. With a producer-side transaction the ambiguity is resolved by the transaction; without one, the publisher cannot know.

Outbound RPCs. A cancelled context on an outbound call propagates only if the transport supports it and the callee honours it. HTTP/2 has RST_STREAM, which terminates the stream and is the standard mechanism; the callee sees the reset when it next writes, not necessarily when it is doing the work. gRPC propagates cancellation this way. A call over HTTP/1.1 has no per-request cancellation, so the only signal is closing the connection, which the callee may not notice until it writes.

Locks and semaphores. A goroutine or thread waiting on a mutex does not observe cancellation, because mutex acquisition has no cancellation variant in most languages. Go’s sync.Mutex has no context parameter. Waiting for a lock on a contended structure can therefore outlast the request by the entire hold time of the current holder, and the waiter has no way to abandon the wait.

The list is not exhaustive, and its length is the point: the set of operations that ignore cancellation includes most of the operations that produce durable effects.

Separating the cancellable from the committed

The design that resolves the ambiguity splits the request into two regions with different scopes.

The first region does the work whose result is worthless if the caller is gone: reading, computing, calling dependencies for data. It runs under the request’s context and stops when that context is done. Wasting this work is a cost, not a correctness problem.

The second region performs the durable effect. It runs under a context detached from the request’s cancellation, with its own short deadline, and it completes or fails as a unit.

// The request context may be cancelled at any moment. The effect must not be.
effectCtx, cancel := context.WithTimeout(context.WithoutCancel(reqCtx), 2*time.Second)
defer cancel()

// idempotency key is fixed before the first attempt and reused on any retry
if err := store.Commit(effectCtx, key, record); err != nil {
	return fmt.Errorf("commit: %w", err)
}

context.WithoutCancel returns a context that carries values but no deadline and no cancellation, which is exactly the semantic needed: the effect should still see the request’s tracing identifiers and tenant, and should not see the caller’s abandonment. The explicit timeout restores a bound so the effect cannot hang indefinitely on a dependency that is itself stuck.

The danger in this construction is that it removes the backpressure that cancellation provided. A dependency that is slow causes every request to enter its effect region, and each effect region holds a slot for its full timeout. Under overload, the effect regions become the unbounded population, and the mechanism that was supposed to bound work becomes the thing that accumulates it. The fix is an admission limit on the effect region specifically, so that a full effect queue fails requests before they reach the commit rather than after.

Idempotency is the other half

Detaching the effect from cancellation means the caller may retry a request whose effect already landed. Without deduplication, that produces two effects.

The standard mechanism is a client-generated key, fixed before the first attempt and carried on every retry. The server records the key alongside the effect, in the same transaction that performs it, and a repeated key returns the original result rather than performing the work again. Recording the key in a separate transaction reintroduces the window: the effect commits, the key does not, and the retry duplicates.

The window that remains is the one between the effect and the key’s durability, which is zero if both are in one transaction and nonzero if the effect is an external side effect. Charging a card and then recording the charge is two systems, and no local transaction spans them. The remedy there is the provider’s own idempotency key, passed through so that the external system performs the deduplication.

The key’s lifetime is a design decision. A key retained forever is an unbounded table; a key retained for an hour covers retries within a request’s plausible lifetime and misses a client that retries the next day. The retention period should exceed the maximum time a client could plausibly spend retrying, which is bounded by the client’s own deadline and retry budget.

Deadlines and cancellation are different signals

A deadline expiring causes cancellation, so the two are often conflated. They carry different information.

A deadline expiry means the request ran out of time. The work may have been progressing normally; the budget was simply too small. The correct handling is to stop and report a timeout, and the effect region should generally not run, because a request that has already exceeded its budget producing a durable effect is a surprise.

A cancellation without a deadline expiry means the caller went away. The reason is unknown: the client may have crashed, the user may have navigated away, or an upstream service may have decided the answer was no longer needed. The work may be worth completing or not, depending on whether the effect is something the system wants regardless.

Separating the two requires checking ctx.Err() rather than treating cancellation as a single condition. Go returns context.Canceled for an explicit cancel and context.DeadlineExceeded when the deadline fired, and a handler that distinguishes them can decide whether to enter the effect region. The distinction is easy to lose when the request context is wrapped by middleware that converts both into the same error.

Measuring the overshoot

The gap between cancellation and the end of work is measurable and worth measuring, because it is the difference between the concurrency a service appears to use and the concurrency it actually uses.

The measurement is the time from the context’s Done closing to the handler returning. A handler that returns within a millisecond of cancellation is checking frequently. A handler that returns seconds later is holding a slot for seconds after its work stopped mattering. Aggregated, that gap is the amount of capacity the service spends on abandoned requests, and it is invisible in every latency metric that stops at the response.

The gap also explains a discrepancy that otherwise looks impossible: a service reporting low latency and low error rates while its CPU is saturated. The saturation is in work whose results were discarded, and no client-visible metric counts it. Queue Time, Service Time, and What a Timeout Measures covers the measurement asymmetry that hides it.

Reducing the overshoot is a matter of check placement. Long loops should check every iteration or every fixed number of iterations. Multi-step handlers should check between steps. Blocking calls should be given contexts or deadlines where the API allows it, and where it does not, the call should be moved into the effect region with an explicit bound rather than left in the cancellable region where it silently ignores cancellation.

Where the split goes wrong

Detaching too much is the common error. A handler that runs its entire body under WithoutCancel has no cancellation at all, and a burst of abandoned requests becomes a burst of full executions. The split should cover only the durable effect, and the effect should be as small as possible: one transaction, one publish, one external call.

Detaching too little is the opposite error and it is more visible: commits fail with cancellation errors, clients see failures for operations that succeeded, and the retry produces a duplicate because the first attempt’s failure was reported before its effect landed. The observable signature is a burst of duplicate side effects correlated with client timeouts.

The third error is a detached region with no bound. A context with no deadline that is not cancelled by anything runs until its dependencies return or fail, and a dependency that never returns holds the region forever. Every detached region needs its own timeout, chosen as the maximum time the effect can legitimately take, and a failure path that reports the effect as unknown rather than as failed. Metastable Failure and the Load That Keeps a System Down describes what happens when regions like that accumulate under load and hold a system in a degraded state long after the trigger has passed.