How strace Sees Every System Call Without Linking Into the Program

A traced program has no tracing code in it. No library is preloaded, no wrapper intercepts the call, and nothing in the binary’s address space changes except, temporarily, the bytes of a breakpoint instruction if the tracer uses one. strace observes system calls because the kernel is already willing to stop a process at the boundary into and out of kernel mode, and because a second process is allowed to inspect and resume it. The mechanism is ptrace, and the shape of the output is dictated entirely by how that mechanism reports state.

The tracee stops twice per system call

The basic protocol is a state machine driven by two processes. The tracer, which is strace itself, calls waitpid and receives notifications when the tracee stops. The tracee stops whenever the kernel decides a tracing event occurred. Between stops it runs normally.

To get a stop at every system call, the tracer resumes the tracee with PTRACE_SYSCALL rather than PTRACE_CONT. That request means “run until the next syscall boundary or signal”. The kernel then arranges a stop at two points: after the process has entered the kernel through the syscall instruction but before the call executes, and again after the call has returned but before control goes back to user space. Each syscall therefore costs two stops, and each stop costs at least one context switch out to the tracer and one back.

At each stop, the tracer reads the tracee’s registers with PTRACE_GETREGS (or the per-thread PTRACE_GETREGSET). The register file is the only channel through which syscall arguments and results are visible, so the architecture’s calling convention becomes part of the output format. On x86-64, the syscall number is in rax at entry, arguments in rdi, rsi, rdx, r10, r8, r9, and the return value replaces the number in rax at exit. On AArch64 the number is in x8 and the return value in x0. This is why a trace of a 32-bit process on a 64-bit kernel can look subtly different: the numbers are from a different table, and the argument registers differ.

Pairing entry with exit is a bookkeeping problem for the tracer, not something the kernel labels. The tracer toggles a per-thread flag on each stop: first stop of a pair is entry, second is exit. Signals complicate the pairing. A signal delivered between the two stops produces an extra stop that is not a syscall boundary, so the tracer must inspect the stop’s signal number to decide whether to toggle. The kernel offers help here: if the tracer sets PTRACE_O_TRACESYSGOOD, syscall stops are reported with SIGTRAP plus 0x80 set in the stop signal, which distinguishes them from a genuine SIGTRAP delivered by the program. Without that option, a debugger breakpoint in the tracee is indistinguishable from a syscall stop, and the toggle drifts out of phase, producing the classic symptom of arguments printed as return values.

A minimal tracer shows the shape:

#include <signal.h>
#include <sys/ptrace.h>
#include <sys/wait.h>
#include <sys/user.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <stdio.h>

int main(void)
{
    pid_t pid = fork();
    if (pid == 0) {
        ptrace(PTRACE_TRACEME, 0, 0, 0);
        execl("/bin/true", "true", (char *)0);
        _exit(127);
    }

    int status;
    waitpid(pid, &status, 0);              /* initial exec stop */
    ptrace(PTRACE_SETOPTIONS, pid, 0, PTRACE_O_TRACESYSGOOD);

    int entering = 1;
    while (waitpid(pid, &status, 0) == pid) {
        if (WIFEXITED(status)) break;
        if (WSTOPSIG(status) != (SIGTRAP | 0x80)) {
            ptrace(PTRACE_SYSCALL, pid, 0, 0);   /* pass other signals on */
            continue;
        }
        struct user_regs_struct r;
        ptrace(PTRACE_GETREGS, pid, 0, &r);
        if (entering)
            printf("enter %lld\n", (long long)r.orig_rax);
        else
            printf("exit  %lld -> %lld\n", (long long)r.orig_rax,
                   (long long)r.rax);
        entering = !entering;
        ptrace(PTRACE_SYSCALL, pid, 0, 0);
    }
    return 0;
}

orig_rax holds the syscall number at both stops; rax holds the number at entry and the result at exit. That duplication exists because the kernel overwrites the register with the return value, and the original number would otherwise be lost.

Why the cost is a multiplier, not a constant

Two context switches per syscall is not a small addition. A syscall that does real work, such as a read that fills a buffer or a fsync, spends most of its time in the kernel, and the tracing overhead is noise. A syscall that does almost nothing, such as getpid, clock_gettime, or the futex calls a runtime makes on every lock acquisition, costs tens of nanoseconds and the trace adds microseconds. Traced workloads therefore slow down in proportion to syscall density rather than uniformly.

The distortion changes behaviour, not just timing. A server that handles a request per syscall will appear to take far longer per request, and any client-side timeout measured outside the traced process will fire. Programs that adapt to latency, through retry timers or backoff loops, will take different code paths under trace. Output that includes timestamps (-t, -tt, -T) makes this visible, but it also invites the mistake of treating those numbers as the program’s real performance.

There is a second reason traces are not neutral. strace calls waitpid, which means the tracer must be scheduled after each stop, and the tracee cannot proceed until it is resumed. On a machine with many runnable threads this converts a parallel workload into a serialised sequence of stop-resume cycles, one per traced thread. The trace is accurate about ordering and misleading about concurrency.

Following threads and children

A single-threaded program is the easy case. ptrace attachment is per-thread: PTRACE_ATTACH on a thread group leader stops only that thread, and its siblings keep running. To trace a whole process, the tracer must enumerate /proc/<pid>/task and attach to each thread individually, or use PTRACE_SEIZE with PTRACE_INTERRUPT, which can stop a group atomically without the side effects of a signal-based attach.

Children are a separate problem. By default a new thread or forked process is not traced, so strace shows the clone call and then nothing from the child. The -f option sets PTRACE_O_TRACEFORK, PTRACE_O_TRACEVFORK, and PTRACE_O_TRACECLONE, which make the kernel attach the new task automatically and report the event to the tracer, which then waits on the new pid as well. -ff writes each traced process to its own output file, with the pid in the name, because interleaving multiple processes into one stream makes the ordering unreadable when they are concurrent.

The automatic-attach path also inherits options, and this is where PTRACE_O_EXITKILL matters. Without it, killing the tracer leaves the tracees stopped in a state where nothing will resume them. With it, the kernel delivers SIGKILL to the tracees when the tracer exits, which is the difference between a lost session and a hung one.

Decoding: where the pretty output comes from

The kernel reports numbers and register values. Everything legible about a trace, such as openat(AT_FDCWD, "/etc/hosts", O_RDONLY) rather than three integers, is produced in the tracer from tables it carries: syscall name and argument-signature tables per architecture, errno names, open flag bits, ioctl request numbers, socket protocol families, and struct layouts for the calls whose arguments are pointers.

Pointer arguments are the interesting case. The tracer reads the tracee’s memory with PTRACE_PEEKDATA (or process_vm_readv) to fetch a pathname string or a sockaddr. Strings are truncated to a fixed limit, 32 bytes by default, adjustable with -s; longer ones print with an ellipsis. That truncation is why a trace of a call with a long path can appear to show a different file than the one actually used. The bytes read are also a snapshot: a buffer the tracee fills concurrently may be read at a moment that does not correspond to any consistent state.

Return values are decoded with a convention that is easy to misread. On Linux, a syscall reports failure by returning a value in the range from -4095 to -1, and the C library converts that to -1 with errno set. strace shows the raw negative value and annotates it, as in -1 ENOENT (No such file or directory). A program that checks for negative returns directly, as some runtimes do, sees the same value the trace shows. A partial success is shown as what it is: read returning 100 when 4096 were requested is a normal short read, not an error, and treating it as one is a common bug that a trace makes obvious.

Filters and what they hide

-e trace= restricts output to a set of syscalls, which reduces both overhead and noise. The filter is applied in the tracer after the stop, so it saves printing and storage but not the stop itself: the tracee still stops twice for every call. Only seccomp-based filtering moves the decision into the kernel, and strace itself does not use it for -e trace=.

Filtering also removes the context that explains a failure. A trace of only write calls shows EBADF with no indication of which descriptor was closed and when. The -y option helps by annotating file descriptors with the path they refer to, read from /proc/<pid>/fd, and -yy extends that to sockets and pipes. Those annotations are themselves a snapshot read through procfs and can fail or change between the syscall and the read, so they are evidence, not truth.

What the trace cannot show

A syscall trace sees the boundary, never the inside. Time spent in the kernel after entry and before return appears as a gap between two lines, not as a breakdown. Library calls that do not cross the boundary are invisible: malloc satisfying a request from its arena, pthread_mutex_lock succeeding on the fast path, a memcpy. A program that appears to do nothing in a trace is doing its work in user space, and the next tool to reach for is a profiler or a sanitizer rather than a longer trace.

The same boundary explains the trace’s strength. Because the kernel is the observer, the record is independent of how the program was compiled, which library it links, and whether it is stripped. The evidence a trace provides is the same evidence the kernel acted on, and that is why it survives as the first tool reached for when a program misbehaves in a way its source does not explain. The register state that strace reads at each stop is the same state a core file preserves at death; the connection between the two is described in what an ELF core file preserves, and the address-to-source translation needed to turn a traced address into a line of code is covered in DWARF line tables.