Restartable Sequences and Interruption Semantics

A thread blocked in a system call has left a partial operation behind. It may have consumed half a packet, taken a lock, or advanced a file offset. A signal arriving at that moment cannot simply interrupt: the kernel has to decide whether the operation can be resumed, abandoned, or completed before the signal is delivered, and the answer differs per call. This is the interruption problem, and its machinery is one of the oldest parts of the system call interface.

Restartable sequences are the modern inversion of it. Instead of the kernel deciding how to resume an operation, user code declares a critical section, and the kernel guarantees that if the section is interrupted by preemption, migration, or a signal, control will be redirected to a handler the program supplied. The program then retries. Nothing is resumed by the kernel, because nothing in the kernel knows what the section was doing.

Both mechanisms exist because a thread can be interrupted at any point, and both are about the same question: what state is left behind, and who is responsible for repairing it.

The exit path is where interruption becomes visible

A system call runs to completion or blocks. If it blocks, it sleeps in an interruptible state, and a signal wakes it. The wakeup sets a per-thread flag indicating that a signal is pending, and the sleep returns an internal error code that is not one of the values a user program ever sees.

The check happens on the way out, in one place: the exit path tests the pending-signal flag and, if it is set, calls the signal delivery code before returning to user mode. That ordering is required. The kernel does not deliver a signal in the middle of a system call; it lets the call reach a point where it can be safely abandoned and delivers on the way out. A call that cannot be abandoned, one that has already committed a change, is not interruptible at all, and a signal arriving during it waits.

For a blocked call the mechanism is a return code convention inside the kernel. The handler returns a negative value in a special range that means “restart” rather than “error”. The range is distinct from the error range and is never visible to user code: the exit path inspects the code, decides whether a restart is permitted, and either re-enters the call with the original arguments or converts the code into the visible EINTR.

The decision depends on three things. Whether the call is restartable at all, since a read that has already transferred bytes is not, and must report the partial count. Whether the signal handler was installed with the flag that requests automatic restart, which POSIX defines and which the C library passes through to the kernel. And whether the signal’s own semantics permit it, since a handler that runs and returns is different from one that terminates the process.

The restart trampoline

Some calls cannot be re-entered with their original arguments, because the arguments are no longer meaningful after partial progress. The clearest case is a sleep: nanosleep interrupted after part of the interval must resume for the remainder, and the remainder is not what the caller passed. Re-entering with the original argument would sleep too long.

The kernel handles this with a dedicated restart code that means “resume whatever was in progress”, and a system call whose only job is to do that. The kernel records enough state to know which call was in progress, and the restart trampoline reads that record and continues. The caller never sees the trampoline; it sees either the completed call or EINTR.

/* What the caller must do when automatic restart is not in effect:
   the partial progress is gone, so the operation is retried whole. */
ssize_t read_all(int fd, void *buf, size_t n)
{
    size_t done = 0;
    unsigned char *p = buf;

    while (done < n) {
        ssize_t r = read(fd, p + done, n - done);
        if (r < 0) {
            if (errno == EINTR)
                continue;               /* retry the remainder */
            return -1;
        }
        if (r == 0)
            break;                      /* end of file */
        done += (size_t)r;
    }
    return (ssize_t)done;
}

The loop is correct only because read reports its partial progress in its return value. A call that returned EINTR after transferring bytes would be unusable in this shape, and the kernel does not do that: it reports the count. This is the contract that makes interruption tolerable at all, since the caller can always tell how much happened.

Where the contract is weaker is on calls that consume a structure rather than a buffer. select and poll modify their arguments in place, and an interrupted call leaves the descriptor sets in a state that the caller must reinitialise before retrying. The kernel does not restore them, and a program that retries without reinitialising polls a set that has already been marked ready. This is a documented hazard rather than a bug, and it is the reason the pattern above does not generalise to every call.

Restartable sequences: the kernel as an abort service

The interruption problem becomes much harder when the critical section is in user space and involves no system call at all.

Consider a per-CPU counter. A thread reads its current CPU identifier, indexes an array, and increments the entry. If the thread is migrated between reading the identifier and incrementing, it increments the wrong entry, and the resulting count is wrong in a way that no amount of locking fixes cheaply, since a lock around the whole sequence costs more than the counter is worth. The window is a handful of instructions, which is exactly the window in which a preemption or a signal can land.

Restartable sequences solve this by having the kernel watch for the interruption and redirect control when it happens. Each thread registers a small per-thread structure with the kernel, and the registration includes a pointer to a descriptor of the critical section currently in progress. The descriptor holds three addresses: where the section starts, how far past the start the commit point lies, and where to jump if the section is aborted.

While the thread executes between the start address and the commit point, the kernel treats it as being inside a restartable region. If the thread is preempted, migrated to another CPU, or has a signal delivered, the kernel sets the return instruction pointer to the abort address. When the thread next returns to user mode, it resumes at the abort handler rather than where it was. The abort handler discards whatever the section had done and jumps back to the start.

/* The descriptor a thread registers. The kernel reads it when it
   needs to know whether an interrupted instruction pointer lies
   inside a section, and where to send the thread if it does. */
struct rseq_cs {
    uint32_t version;
    uint32_t flags;
    uint64_t start_ip;              /* first instruction of the section */
    uint64_t post_commit_offset;    /* bytes from start_ip to the commit store */
    uint64_t abort_ip;              /* where control lands on an abort */
};

The section itself must satisfy one rule above all: its only memory side effect is a single store at the very end, so that an abort can discard it by doing nothing. Placed against the descriptor, the section takes the shape below. The register names and mnemonics are schematic, standing for whichever instruction set the section is written for; what matters is the placement of the single store, the commit point immediately after it, and the abort label placed so that falling into it is harmless:

    mov  r1, [cpu_slot]        /* read this thread's CPU id */
    lea  r2, [slots + r1*4]
1:  add  dword [r2], 1         /* section start */
2:                             /* commit point: the store above */
    jmp  4f
3:                             /* abort lands here, having done nothing */
4:  /* ... */

The descriptor’s field order and the offset semantics are part of the ABI, not something a program should construct by hand; the real interface is reached through a library that emits the assembly and the descriptor together. The shape is what matters: a section whose only side effect is at the very end, so that aborting it is always safe, and an abort path that re-reads the CPU identifier before trying again.

The commit point is the subtle part. Everything before it must be undoable by simply not having happened; everything at it must be a single store. That is why the sequence ends with one write and not with a read-modify-write loop: an atomic operation that might have partially completed cannot be aborted cleanly, and a store that has not retired has not happened.

What the abort path guarantees, and what it does not

The guarantee is narrow and precise: if the thread is descheduled, migrated, or signalled while its instruction pointer is inside the registered section, the kernel will redirect it to the abort address before returning to user mode. It does not redirect for other reasons, and it does not preserve any register state from the section.

Three consequences follow. The abort handler must be able to run with whatever register values the section left, so the compiler must not assume the section completed. The retry must re-read everything the section depended on, because the reason for the abort may have been a change to exactly that. And the section must not have side effects outside its own memory, because an abort does not undo them.

The registration itself is per-thread and does not survive exec; a program that replaces its image must register again, and a library that relies on the mechanism cannot assume a registration it did not make. The kernel does not validate the addresses beyond checking that they are in the user range, so a program that registers a bad abort address gets a crash rather than an error.

There is also a cost that shows up in the kernel’s own fast paths. Because a registered thread can be inside a section at any moment, the preemption path must check whether the outgoing thread has a registration and whether its instruction pointer is in range. That check runs at every context switch, which is the price of the mechanism: a small addition to every switch, paid by all threads, to make a handful of instructions safe for the ones that use it.

Where the two mechanisms meet

A restartable sequence and an interrupted system call are the same problem seen from opposite sides. In both cases a thread’s progress is broken by an event it did not cause, and in both cases the repair is defined by a contract rather than by the kernel’s understanding of what the thread was doing.

The difference is who holds the state. For a system call the kernel holds it, so the kernel can restart or report progress precisely. For a user-space section the kernel holds nothing but an address, so the only repair it can offer is a jump to a handler the program wrote. That is less capable and vastly cheaper, and it is enough for the case that motivated it: keeping per-CPU data consistent without locks on the hot path.

The cluster’s recurring theme applies to both. The boundary between kernel and program is expensive per crossing, so mechanisms either reduce crossings, as io_uring and Shared-Memory Submission Queues does, or make the crossing carry more meaning. Restartable sequences go further and make the kernel into a service that user code never calls, but always depends on. The price is a contract with sharp edges: the section must be abortable, the handler must be reentrant with respect to whatever interrupted it, and the retry must be correct when the abort was caused by a signal that will be delivered the moment the handler returns.

That last point is the one that catches implementations. A signal delivered during a section causes an abort, the handler runs, and the signal handler itself may run before the retry, and the signal handler may change the very state the section was updating. The section’s retry must therefore be correct not just after a migration but after arbitrary user code has run, which means the state it reads must be re-validated rather than assumed. The same discipline that makes a system call wrapper correct, described in Error Returns and the errno Convention, applies here in a harsher form: every failure must be anticipated by the code that resumes, because nothing else will repair it.