Thread Cancellation, Cleanup Handlers, and Deferred Delivery

Killing a thread is easy to specify and impossible to implement safely. A thread can be stopped at any instruction, and at that instruction it may hold a lock, have a partially updated data structure, or be in the middle of a library call that expects to complete. Nothing in the machine state says which of those conditions holds, so a mechanism that stops a thread without its cooperation cannot avoid leaving the process inconsistent.

Cancellation Is a Request

The thread interface therefore defines cancellation as a request that the target thread acts on at a point of its own choosing. The call that makes the request returns immediately with an indication that the request was recorded. The target continues to run until it reaches a point where it is willing to be cancelled, at which point it terminates.

Three properties follow from that definition and each one is a source of confusion.

First, the request has no effect on a thread that never reaches a cancellation point. A compute loop that makes no library calls and no system calls will run to completion regardless of any number of requests. This is not a bug in the request; it is the only safe behaviour, because the alternative is termination at an arbitrary instruction.

Second, the request is not a signal. It is recorded in the thread’s control block as a flag and as a count of pending requests for the deferred form. It does not interrupt a blocking call, though the point where that call returns may be a cancellation point that observes the flag.

Third, the request is not a priority. A thread that is cancelled while blocked in a call at a cancellation point is unblocked and terminated, which is a stronger action than merely waking it. The call does not return a normal result; the thread runs its cleanup handlers and exits.

The type of cancellation determines when the request takes effect. The deferred type is the default and the only one that is safe in general: the request is acted on at the next cancellation point. The asynchronous type acts immediately, at any instruction, and is specified as usable only for threads that do not hold resources and do not call functions that are not async-cancel-safe. In practice that means a pure compute loop with no locks and no allocation, which is a narrow category.

What Counts as a Cancellation Point

A cancellation point is a function that the standard lists as one, or a function that may block and is documented to be one. The list is long and includes the calls that wait for input or for a condition: reading from a descriptor, waiting on a condition variable, joining another thread, sleeping, and accepting a connection. The rationale is that these calls already have a point at which the thread can be interrupted and its state made consistent, so adding cancellation to that point costs nothing extra.

The list also includes some calls that do not block, such as writing to a descriptor, because the write may block on a full buffer. The classification is by possibility rather than by typical behaviour, which is why the list is not intuitive.

Two calls are deliberately not cancellation points: the one that reads a line from a stream and the one that reads a formatted record, because they may have already consumed part of the input and cannot be restarted. A thread cancelled inside them would leave the stream in an unknown position, and the standard’s authors chose to protect the stream rather than to make the call cancellable.

A thread can also create its own cancellation points. A function that acquires a resource, does work, and releases it can call the test function at a point where its state is consistent, and the test function will act on a pending request there. This is how a long-running loop becomes cancellable without being broken into blocking calls.

The interaction with signal masks is a subtlety. A cancellation request does not interact with the signal mask, because it is not a signal. A thread that has blocked every signal can still be cancelled at a cancellation point, and a thread that has unblocked everything is not made more cancellable by that. The two mechanisms are separate, which is why the reaping and delivery rules in Signal Delivery, Thread Masks, and Which Thread Runs the Handler do not apply to cancellation.

Cleanup Handlers as a Scoped Stack

The resource problem is that a cancelled thread may hold a lock, have an open file, or have allocated memory. Since the thread is terminated at a point the library chooses, the code after the cancellation point never runs, so an ordinary goto cleanup or a scope exit does not help.

The mechanism is a stack of cleanup handlers per thread. A handler is a function and an argument, pushed before a resource is acquired and popped after it is released. When the thread is cancelled, the library runs the handlers from the top of the stack down, in reverse order of installation, before terminating the thread. A handler that unlocks a mutex, closes a descriptor, or frees a block therefore runs even on the cancellation path.

The push and pop operations are usually macros rather than functions, because they must establish a scope in the calling function. The push opens a block and the pop closes it, so the pair must appear in the same function and the pop must be lexically after the push. This is why a cleanup handler cannot be registered in one function and removed in another, and why a function that returns from between the two macros must use a form that pops the handler before returning.

#include <pthread.h>

static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t ready = PTHREAD_COND_INITIALIZER;
static int pending = 0;

static void unlock(void *arg) {
    pthread_mutex_unlock((pthread_mutex_t *)arg);
}

static void *worker(void *arg) {
    pthread_mutex_lock(&lock);
    pthread_cleanup_push(unlock, &lock);   /* runs if cancelled below */

    while (!pending) {
        pthread_cond_wait(&ready, &lock);  /* a cancellation point */
    }

    pthread_cleanup_pop(1);                /* 1 = run the handler now */
    return NULL;
}

The count argument on the pop decides whether the handler runs on the normal path. Passing a nonzero value runs it, which lets one handler serve both the normal and the cancellation path. Passing zero removes it without running it, which is correct when the code after the pop has already released the resource.

The handlers run on the thread’s own stack, at the point where cancellation was observed. They therefore have the same limited stack space the thread had, and a handler that allocates a large object can overflow; the sizing and guard-page arrangement of that stack is described in Thread Stacks, Guard Pages, and Thread-Local Storage. They also run with the cancellation state set so that a further request does not re-enter them, which prevents a handler from being cancelled while cleaning up.

The Deferred Request and the Join

A cancelled thread does not simply disappear. It exits through the same path as a normal return, which means it remains in a state where its result can be collected, and a join by another thread returns the cancellation indication. This is what allows a program to cancel a worker and then join it, with the guarantee that the worker’s cleanup has completed before the join returns.

The alternative, detaching the thread and cancelling it, removes the ability to know when cleanup finished. A program that closes a resource immediately after issuing a cancel request races the handler, and the handler may then operate on a closed descriptor or a freed block. The safe pattern is cancel-then-join, and the cost is that the canceller must be willing to wait for the target to reach a cancellation point.

A thread that never reaches a cancellation point makes the join wait forever, which is why the deferred design requires the target to cooperate. A loop that does long computation must call the test function periodically, and a loop that waits on a condition must use a wait that is a cancellation point rather than a spin. Without that cooperation, cancellation is a request that is never honored.

The cleanup stack is also unwound by a normal return from the start function and by an explicit thread exit. The library runs the handlers in all three cases, so a function that pushes a handler and returns normally without popping it has a bug that manifests as a double release rather than as a leak.

Failure Modes

The most common defect is a cleanup handler that calls a function which is not async-cancel-safe, typically one that takes a lock or allocates. The handler runs in a context where the thread has been interrupted at an arbitrary point, and a lock it tries to acquire may be held by the very operation that was interrupted. The deadlock that results is indistinguishable from a normal lock-order problem except that it appears only after a cancellation.

The second is a handler that is popped without running while the resource is still held. This happens when the code between push and pop returns early through a path that does not pop, or when the pop is written with a zero argument on a path that has not released the resource. The leak is silent until the process exhausts descriptors or the mutex stays locked and another thread blocks forever.

The third is cancellation of a thread that holds a lock the canceller needs. The canceller issues the request, the target is at a cancellation point inside a critical section, the handler unlocks correctly, and the sequence works. If the handler is missing, the lock stays held and the canceller deadlocks on its next acquisition. This is the case the cleanup mechanism exists for, and the case where its absence is most expensive.

A fourth is specific to the asynchronous type. A thread cancelled asynchronously in the middle of a library call leaves the library’s internal state inconsistent, because the call had no chance to complete or to clean up. The standard restricts asynchronous cancellation to code that makes no such calls, and the restriction is not enforceable by the library, so a program that violates it gets corruption rather than an error.

The design conclusion is that cancellation is a cooperative protocol, and like every cooperative protocol it depends on the participants reaching the points where they have agreed to be interrupted. The parts of the process that can be relied on are the cancellation points the standard lists and the cleanup handlers that run when one is observed. Everything else is a request that may never be honored, and a program that treats it as a kill has misunderstood the interface.