ptrace and the Stop-Notify Model Behind a Debugger

A debugger does not run the program. It asks the kernel to make the program stop at interesting moments, and then it reacts. The interface for that request is ptrace, and its shape is unusual: it is not a callback interface and not a shared-memory protocol. It is a set of one-shot commands issued by a tracer thread against a tracee thread, plus a rule that the tracee can only make progress when the tracer tells it to. Everything a debugger does, from breakpoints to stepping to reading memory to inspecting registers, is built on that asymmetry.

Attaching is a signal, not a handshake

A process becomes traceable in one of two ways. It can call PTRACE_TRACEME on itself before executing a new image, which is what a debugger does when it launches a program: the child asks to be traced, then calls execve, and the resulting SIGTRAP after a successful exec is the first stop. Or an existing process can be attached from outside with PTRACE_ATTACH, which stops the target as if it had received SIGSTOP and leaves the tracer waiting for that stop.

PTRACE_SEIZE is the newer form. It establishes the tracing relationship without stopping the target and without sending a signal, so a tracer can set options first and then request a stop with PTRACE_INTERRUPT. The distinction matters for tools that attach to a running server: PTRACE_ATTACH interrupts it whether or not the tracer is ready, while PTRACE_SEIZE lets the tracer install its event mask before the first stop arrives.

Attachment is per thread, not per process. A multi-threaded target has one task per thread, and each must be attached; the kernel does not extend an attachment to threads created later unless the tracer asks for it with an option that reports clone events. A tracer that attaches to a process identifier and then waits will see only the thread whose identifier matches, which is why tracers enumerate /proc/<pid>/task and attach to each entry.

Only one tracer may be attached to a given thread at a time. A second PTRACE_ATTACH fails with EPERM, and permission for the first is governed by the usual rules: the tracer must be the parent, must share the target’s user identifier, or must hold the capability that bypasses the check. A system-wide policy knob can further restrict attachment to descendants, which is why a debugger started from a shell can attach to the shell’s children but not to an unrelated process.

Every stop is a message the tracer must answer

The tracee stops in the kernel and stays stopped. The tracer learns of it through waitpid, with the __WALL flag so that thread stops are reported as well as process exits. The wait returns a status word that says whether the tracee stopped, and with which signal.

The important part is what happens next: the tracee does not resume on its own. The tracer must issue a resume request, which is PTRACE_CONT to continue, PTRACE_SINGLESTEP to execute one instruction, or PTRACE_SYSCALL to stop at the next system call boundary, and that request carries a signal number. If the number is zero, the stop is consumed and nothing is delivered. If it is nonzero, that signal is delivered to the tracee as it resumes.

That single parameter is the whole mechanism by which a breakpoint works. The int3 byte raises SIGTRAP; the tracer sees a stop with WSTOPSIG(status) == SIGTRAP, decides that this is its own breakpoint, adjusts the program counter, restores the byte, and resumes with a signal argument of zero. The SIGTRAP is swallowed. A debugger that resumed with SIGTRAP instead would deliver the signal to the program, and the program would either die or run a handler that has no business running.

#include <sys/ptrace.h>
#include <sys/wait.h>
#include <signal.h>

int status;
pid_t pid = /* tracee */;

waitpid(pid, &status, __WALL);
while (WIFSTOPPED(status)) {
    int sig = WSTOPSIG(status);
    int event = (status >> 16) & 0xff;     /* 0 unless an option is enabled */

    if (sig == (SIGTRAP | 0x80)) {
        /* syscall stop, enabled with PTRACE_O_TRACESYSGOOD */
    } else if (sig == SIGTRAP && event == 0) {
        /* breakpoint or single-step */
    }
    /* Pass 0 to swallow the signal, sig to deliver it. */
    ptrace(PTRACE_CONT, pid, 0, (void *)(long)(sig == SIGTRAP ? 0 : sig));
    waitpid(pid, &status, __WALL);
}

The same rule applies to signals the program legitimately receives. If the tracee is sent SIGUSR1 while stopped, the kernel does not deliver it; it reports a stop instead, and the tracer decides. Passing SIGUSR1 back delivers it at the point of resume, which is almost but not exactly where it would have arrived untraced. This is why attaching a debugger can change the timing of signal-driven programs, and why a tracer that ignores stops changes program behaviour rather than merely observing it.

Telling stops apart

Every stop arrives as a signal number, and several distinct events share the same number. Disambiguating them is most of a tracer’s logic.

A breakpoint, a single-step completion, and an event notification all arrive as SIGTRAP. The first two are distinguished by the program counter and by the debug status register on architectures that have one; the third by the high byte of the wait status, which carries an event code when the tracer has enabled the corresponding option. Fork, clone, exec, and exit events each have their own code, and the identifier of the new child is available through a query on the stop.

A system call stop is a SIGTRAP as well, which is a problem: a tracer using PTRACE_SYSCALL to watch system calls cannot tell a syscall stop from a breakpoint by the signal number alone. The PTRACE_O_TRACESYSGOOD option changes the reported signal to SIGTRAP | 0x80, a value that cannot be a real signal, and the ambiguity disappears. Distinguishing entry from exit within a syscall stop requires either the newer syscall-info query or a heuristic on the return-value register, which holds a negated error number on entry and a result on exit.

A group stop is the third case. When the tracee receives SIGSTOP, SIGTSTP, SIGTTIN, or SIGTTOU, the whole thread group stops, and the tracer sees a stop whose signal is that number. It is not a signal-delivery stop: there is no signal to suppress or deliver, and resuming with PTRACE_CONT does not necessarily restart the group. The documented way to distinguish the two is to ask for the stop’s signal information; a group stop has none, and the query fails. Tracers that conflate the two either lose job control or hang a stopped process.

What the model costs

Each stop is two context switches, one into the tracer and one back, plus whatever work the tracer does while the tracee is frozen. For a breakpoint that fires once, the cost is irrelevant. For PTRACE_SYSCALL on a program that makes a system call per iteration, the cost is the entire runtime: the tracee executes almost nothing between stops, and the tracer’s per-stop work dominates.

The design also imposes a constraint on the tracer’s own code. While handling a stop, the tracer may read the tracee’s memory and registers, but it must not block indefinitely, because the tracee cannot make progress until the tracer issues a resume. A tracer that waits on a lock held by another thread which is itself waiting on the tracee has created a deadlock the kernel will not break. Debuggers avoid this by keeping the event loop single-threaded with respect to a given tracee, or by ensuring that any thread that might block is not the one holding the tracing relationship.

Resuming is also where most of the subtle races live. Between the tracer’s decision to resume and the tracee’s actual resumption, another thread in the tracee’s process may run and reach the same breakpoint address. All-stop mode prevents this by keeping every thread stopped while one is resumed, at the cost of making the program’s own concurrency invisible. The alternative, letting threads run independently, is what per-thread stop control exposes over a remote connection, and it requires the tracer to track a stop state per thread rather than per process. The same resume path carries the trap flag when a single instruction step is requested, so the tracer’s next stop arrives after exactly one instruction rather than at the next breakpoint.

None of this is visible in the debugger’s user interface, which presents a single notion of “the program is stopped”. The kernel’s model is finer-grained and more literal: threads stop individually, stops are messages, and messages must be answered.