Cancellation and Drop at Suspension Points

A task can be stopped in two ways. It can be asked to stop, which requires it to cooperate at a point where it can clean up, or it can be made to stop, which means destroying its state wherever it happens to be. The first is cancellation as most runtimes implement it; the second is what a process kill does, and it is not available to a runtime that shares an address space with other tasks.

The interesting territory is between them. A runtime can destroy a suspended task cheaply, because a suspended task is not running and its state is inert. The difficulty is that “suspended” is a property of a particular moment, that the moment is chosen by whoever cancels, and that the state at that moment may include resources the task acquired and has not yet released. Where that moment can occur at all is decided by the coroutine representation, as stackful and stackless coroutines explains.

Cancellation is a request delivered at a suspension point

The standard shape is a flag plus a check. A task’s handle carries a cancellation flag; the runtime sets it; the next time the task reaches a suspension point, the runtime observes the flag and completes the task with a cancellation outcome instead of resuming it. Nothing happens between suspension points, so a task in a tight computation continues until it reaches one.

This design is the only one that is sound for a stackless runtime, and it has a clear semantics: the task is cancelled at a specific await, and the state at that await is well defined. It also means cancellation latency is bounded by the distance to the next await, which is the same bound as scheduling latency and fails in the same way: a task that never awaits is never cancelled.

Delivering the request at a suspension point also means the cancellation can be observed by the code that awaited. An await that returns “cancelled” is a value the task can handle; it can roll back, release what it holds, and return an outcome. The alternative, where the runtime destroys the frame without letting the task run, leaves no opportunity for cleanup except destructors.

What destructors can and cannot do

In a language with deterministic destruction, dropping a suspended frame runs the destructors of every live local. That is the mechanism that makes cancellation tolerable: the frame holds the task’s resources, and destroying the frame releases them in the reverse order of acquisition.

The mechanism has limits that are worth stating precisely. A destructor runs synchronously, on the thread that drops the frame, and it cannot suspend. A destructor that needs to perform an asynchronous operation, whether flushing a buffer to a socket, committing a transaction, or sending a message, cannot await, because the drop is happening inside the runtime’s own bookkeeping and the runtime is not in a position to suspend the task being destroyed. Runtimes that need this provide a separate shutdown path: the task is resumed with a cancellation signal, performs its cleanup asynchronously, and only then completes.

The second limit is that a destructor can panic or fail, and a failure during unwinding is a double fault. Most languages that use unwinding for cleanup treat a panic during unwinding as fatal, which means a resource whose destructor can fail makes cancellation unsafe. The workaround is to make the destructor infallible, or to catch the failure and record it, which pushes the problem to whoever reads the record.

The third limit is that not all state is owned by the frame. A task may have handed a buffer to the kernel, registered a descriptor with the event loop, or placed an entry in a shared queue. Dropping the frame releases what the frame owns; the registrations and the shared entries need explicit removal, and the task’s destructor is the only place that knows to do it. A runtime that cancels a task without running its destructor leaves the event loop holding a registration for a task that no longer exists, and the symptom is a wakeup that dispatches to freed memory.

Partial operations and the ambiguity of “did it happen”

The hardest case is a cancellation that arrives while an operation is in flight. A task that has written half a request, or submitted a completion that the kernel has not yet reported, is in a state where the operation’s effect is unknown.

For a write, the ambiguity is resolvable by the protocol: a cancelled task that has written a partial request must either complete it or close the connection, because a peer that receives half a request will wait for the rest. Closing is the usual choice, and it makes cancellation visible to the peer as a truncated message rather than as a silent stall.

For a completion-based operation, the ambiguity is not resolvable by the task. The request was submitted, the kernel owns the buffer, and the completion will arrive whether or not the task still exists. The runtime must therefore keep the completion’s tag alive until the entry is reaped, and must not free the buffer until then. This is the reason completion-based runtimes have a separate bookkeeping layer for in-flight operations, distinct from the task table: an operation can outlive its task, and the runtime must handle the case where a completion arrives for a task that was cancelled two seconds ago.

struct in_flight {
    unsigned char *buf;
    size_t len;
    uint64_t tag;              /* copied into the completion entry */
    task_id_t owner;           /* may be cancelled before the entry arrives */
};

/* Called once per reaped completion, whatever happened to the task. */
void on_complete(struct in_flight *op, long res)
{
    task_t *t = op->owner ? task_lookup(op->owner) : NULL;
    if (t)
        task_wake(t, res);          /* res is bytes moved, or -errno */
    free(op->buf);                  /* the buffer belongs to the operation */
    free(op);
}

The pattern generalises: any resource whose lifetime is tied to an operation rather than a task must be reclaimed by the operation’s completion, not by the task’s destruction. Runtimes that get this wrong produce leaks that only appear under cancellation, which is why cancellation paths deserve the same testing as success paths.

Structured cancellation and the scope tree

A task cancelled while it has spawned children must decide what happens to them. The two policies are to cancel the children too, or to let them run. The first is what structured concurrency prescribes: a task’s lifetime encloses its children’s, so a cancelled parent cancels its subtree, and a parent cannot complete until its children have completed or been cancelled.

The implementation is a tree of scopes with a cancellation flag propagated down. Cancelling a scope sets the flag on every descendant, and each descendant observes it at its next suspension point. The parent waits for all descendants to acknowledge, which means the cancellation’s latency is the maximum over the subtree rather than the latency of one task. That maximum can be large, and it is unbounded if any descendant ignores the flag.

The alternative, detached children, makes cancellation local and leaves the children running with no one to observe their failure. Detached tasks that fail silently are a common source of resource exhaustion, because the failure is recorded nowhere and the task’s resources are held until it happens to finish.

A middle design is a grace period: the scope signals cancellation, waits for a bounded interval, and then abandons whatever remains. Abandonment requires the runtime to be able to reclaim a suspended task’s resources without its cooperation, which brings back the destructor question: if a task holds a lock or a registered descriptor, abandoning it leaves that resource held. Runtimes that offer this usually restrict it to tasks that hold nothing but memory.

Timeouts are cancellation with a clock

A timeout is a cancellation whose trigger is a timer rather than a caller. The mechanism is identical: a flag, a check at the next suspension point, and a cancellation outcome. The difference is that the timer must be armed somewhere, and arming a timer per operation is expensive if the timer facility is a sorted list of absolute deadlines.

The standard implementation is a hierarchical timing wheel: a set of buckets indexed by the low bits of the deadline, with each level covering a coarser range, so that insertion and removal are O(1) and expiry is a scan of the current bucket. The wheel’s resolution is the granularity of the lowest level, so a timeout is accurate to that granularity and not better. A runtime with a ten-millisecond lowest level cannot honour a one-millisecond timeout precisely; it can only guarantee that the task is cancelled no earlier than the deadline and no later than the deadline plus the resolution.

The interaction with cancellation is that a timeout’s timer must be cancelled when the operation completes normally. A runtime that leaves expired timers in the wheel accumulates entries for operations that finished long ago, and the wheel’s memory grows with the number of operations rather than the number of pending ones. Removing a timer from a wheel requires a reference to its entry, so the runtime must keep one, which means the operation’s state includes a timer handle that must be released on every path. The timer wheel’s design is taken up in hierarchical timing wheels for timeouts.

The invariant to preserve

The rule that makes cancellation tractable is that a task must be in a consistent state at every suspension point. Concretely: every resource it holds is reachable from its frame or from a table keyed by its identity, and every operation it has started is either completed or tracked independently of the task.

A task that violates the rule at one await, taking a lock, awaiting, and releasing it, is correct only if the lock is released on the cancellation path too. A task that registers a descriptor, awaits, and deregisters is correct only if the deregistration is in a destructor rather than in the code after the await. These are not exotic requirements; they are the same requirements that apply to exceptions, and the reason cancellation is often described as asynchronous exceptions is that it has the same shape and the same failure modes.

The difference is that an exception propagates up the stack immediately, while cancellation may be delivered at any of a task’s awaits, including ones the programmer did not consider. That makes the discipline harder to maintain and makes tooling more valuable: a runtime that can report which tasks were cancelled at which await, and which resources they held, turns a class of intermittent leaks into a diagnosable pattern. The fairness machinery in fairness and tail latency in task schedulers records similar per-task state, and the two are usually implemented in the same accounting layer.