Cancellation Points and the Half-Finished Cleanup State

Every cleanup mechanism in the earlier articles assumes that the code between acquisition and release runs to a point of its own choosing. Cancellation breaks that assumption. A thread or task can be told to stop at a moment when it holds a lock, has written half a record, and is three steps into a five-step cleanup. The resource question then becomes a question about reentrancy: whether the cleanup path can be interrupted by the same cancellation that invoked it, and whether running it twice is safe.

There are two families of cancellation, and they produce different failure modes.

Asynchronous cancellation and its two windows

POSIX threads offer deferred and asynchronous cancellation. Deferred cancellation takes effect only at designated cancellation points, which are the blocking calls: read, write, wait, sleep, accept, connect, and the functions that wait on condition variables or join threads. Asynchronous cancellation takes effect at any instruction. The first is manageable; the second is nearly impossible to use correctly, and the POSIX rationale recommends asynchronous cancellation only for functions that are async-cancel-safe, which in practice means code with no cleanup obligations.

The reason is that asynchronous cancellation can land in the middle of a sequence that must be atomic from the perspective of cleanup. If the thread is cancelled after taking a lock but before recording that it took it, the cleanup handler cannot know whether to release the lock. If it is cancelled in the middle of an allocator’s free-list update, the allocator’s own invariants are broken and the process is corrupted in a way no cleanup can repair.

Deferred cancellation still has windows, but they are placed deliberately. The rule is that a function which acquires a resource and then calls a cancellation point must register a cleanup handler first, so that the cancellation unwinds through it. POSIX provides a stack of handlers per thread, pushed with pthread_cleanup_push and popped with pthread_cleanup_pop, and the handlers run in reverse order of registration when the thread is cancelled or when it exits the scope normally with a nonzero execute flag. That stack is the same LIFO discipline described in release order at scope exit, applied to a stack maintained by the threading library rather than by the language.

pthread_mutex_lock(&m);
pthread_cleanup_push(unlock_guard, &m);

while ((n = read(fd, buf, sizeof buf)) > 0) {   /* a cancellation point */
    process(buf, n);
}

pthread_cleanup_pop(1);                          /* unlock on the normal path */

The macro pair is required to be lexically paired in the same block because the implementation may expand them into a brace-enclosed scope. That constraint is a symptom of the design: the handler stack is a language-level construct implemented with macros, and it does not compose with return statements that jump out of the block.

What a cleanup handler may call

The handler runs in the cancelled thread, so it can take locks and free memory. The constraint is different from the signal-handler constraint and is worth separating, because the two are often confused. A cancellation handler is ordinary code running in ordinary thread context; it can call free, close, and pthread_mutex_unlock. A signal handler runs in a context that interrupted arbitrary code, and may only call the small set of functions that POSIX lists as async-signal-safe.

The signal case is the one that breaks resource management outright. malloc and free are not async-signal-safe, because the interrupted thread may have been inside the allocator holding its internal lock. A signal handler that calls free can deadlock against the thread it interrupted. The same applies to printf, which locks the stdio stream, and to any function that is not on the list.

The safe pattern for a signal handler is to do nothing but set a flag of type volatile sig_atomic_t or write a byte to a pipe that the main loop reads, and to let the main loop perform the cleanup. That is the self-pipe trick, and it converts an asynchronous event into a synchronous one at a point the program chooses. The cost is a descriptor pair and a wakeup, and the benefit is that all the resource logic runs in ordinary context where locks and allocation are legal.

Cooperative cancellation and the state it leaves

Languages with their own runtimes generally avoid asynchronous cancellation entirely. Go’s context package is cooperative: a context carries a Done channel that is closed when the context is cancelled, and functions that may block select on that channel alongside their work. Nothing forcibly interrupts the goroutine. The goroutine notices the closure at a point it chose, and then runs its deferred cleanup normally.

The design puts the burden on the code that does the blocking. A loop that reads from a connection must select on the context, and a library that performs a long computation must check periodically. A function that ignores the context runs to completion, and the cancellation is delayed until it returns. That is a real limitation and the reason context plumbing appears in every function signature along a request path.

The advantage is that the cleanup path runs with the same guarantees as any other code. Deferred calls run in LIFO order, no handler is interrupted, and there is no window in which a lock is held by a cancelled thread. The disadvantage is that a goroutine blocked in a syscall the runtime cannot interrupt will not observe the cancellation until the syscall returns.

What cooperative cancellation does not solve is the half-finished external state. A request cancelled after it has written half a response body, or after it has inserted a row into one of two tables, leaves a state that no in-process cleanup can undo. The cleanup must therefore be written as compensation, with each step recording what it has done so that the inverse can be applied. That is the same commit-boundary reasoning as failure atomicity on the error path, with cancellation as an additional source of early exit.

Cancellation while holding a lock

The most consequential case is a lock held across a cancellation point. In the deferred model the handler releases it, so the design is sound as long as the handler is registered immediately after the lock is taken. In the asynchronous model the same code is unsound, because the cancellation can land between the pthread_mutex_lock return and the handler registration.

In the cooperative model the question is different: a goroutine that holds a mutex and then waits on the context’s Done channel will release the mutex when it runs its deferred unlock, provided the wait is inside the same function scope. A goroutine that holds a mutex and blocks on a channel with no cancellation case holds the mutex for the duration, and the cancellation does not help.

Rust’s approach is structural. A mutex guard is a value that unlocks on drop, so the lock cannot be held across a cancellation point without the guard being in scope. An async runtime that cancels a future drops it at an await point, and dropping the future drops the guard, which unlocks. The guarantee is that the guard’s destructor runs; the caveat is that a future can be dropped at any await, so state that must be updated atomically across an await needs to be in a form that can be dropped safely at any of those points.

The reentrancy rule for cleanup

The property every cleanup path needs, whatever the cancellation model, is that it can be entered when the operation is partially complete. Three concrete requirements follow.

The first is that each acquisition records its own completion before the next step begins. A structure that sets a opened flag only after both the open and the initial read succeed cannot distinguish the state where the open succeeded and the read failed, so the cleanup does not know whether to close. Two flags, one per step, cost two bits and remove the ambiguity.

The second is that the release be idempotent. A cleanup that runs from a cancellation handler and again from a normal exit path must not release twice. Setting the handle to an invalid value after the release, and testing for that value at entry, is the standard way, and it is the same null-out rule that makes a moved-from object destructible in ownership transfer.

The third is that the cleanup must not itself be cancellable at a point where it would stop halfway. POSIX provides a cancellation state and type per thread: PTHREAD_CANCEL_DISABLE postpones cancellation until it is re-enabled. Wrapping the cleanup in a disable/enable pair is the standard technique, and it is why a handler that releases several resources is written as one unit rather than as several handlers.

Where the accounting lives

Cancellation exposes a gap that deterministic cleanup alone does not: the resource may be held by an entity that is no longer running any code. A thread that has been cancelled is gone, but a descriptor it opened is still open until something closes it, and a lock it held is still held unless its handler released it. The operating system reclaims descriptors and memory at process exit, which is why a crashed process does not leak in any lasting sense, but a cancelled thread in a long-lived process is not a process exit.

The accounting has to be external to the thread for that reason. A registry that records which resources belong to which unit of work, and a supervisor that releases the registry when the unit is cancelled, gives the property that in-thread handlers give within a thread, extended across threads. That registry is also the natural place to implement the draining described in shutdown ordering, where the unit of work is the whole process and the reclamation is done by the kernel.