Signal-Driven Sampling: How a Profiler Captures a Stack Without Instrumenting Code

Instrumenting every function entry to count calls and accumulate time produces exact numbers and a program that runs measurably slower. A call counter on a hot function costs a load, an add, and a store; inlined into a loop that executes a hundred million times a second, that cost is no longer noise. Profilers avoid the problem by not measuring every event. They interrupt the program at a fixed frequency, record where the instruction pointer happens to be, and let arithmetic over many samples stand in for the exact accounting.

The consequence is that a profile is a statistical estimate, not a ledger. That distinction is not a philosophical one: it determines how many samples are needed before a number is meaningful, and it is why the delivery path of the interrupt matters as much as the frequency.

Why the interrupt must come from outside the program

A sampling profiler needs to observe a thread at an arbitrary instant without the thread’s cooperation. There are two ways to arrange that. The program can call into the profiler itself at chosen points, which is instrumentation by another name and inherits its blind spots: code that never reaches the instrumentation point is never sampled. Or the operating system can interrupt the thread, which is the approach every low-overhead profiler uses.

On Linux the oldest interface is setitimer(ITIMER_PROF, ...). The kernel keeps a per-process interval timer that decrements while the process consumes CPU time, in both user and system mode, and delivers SIGPROF each time the interval expires. Because the timer is charged against CPU time rather than wall time, a process that sleeps for a second accumulates no ticks and produces no samples, which is exactly right for a CPU profiler and exactly wrong for anything interested in waiting.

Newer code uses POSIX per-process timers: timer_create with a clock id, a sigevent describing which signal to send, and an interval. The clock choice changes the meaning of the sample. CLOCK_PROCESS_CPUTIME_ID charges against the whole process; CLOCK_THREAD_CPUTIME_ID charges against one thread, and combined with a thread-directed delivery the timer fires in the context of the thread it is measuring. The third route bypasses signals entirely: perf_event_open programs a hardware performance counter to overflow after a fixed number of events, and the kernel delivers samples directly.

The distinction between a process-directed and a thread-directed signal is where most home-grown samplers go wrong. SIGPROF raised by setitimer is process-directed. The kernel picks any thread in the process that does not block it, which on a process with sixteen worker threads means the samples are spread across threads by the scheduler’s choice rather than by measurement intent. A thread that spends its life in a tight compute loop may be interrupted rarely if the signal keeps landing on its siblings. Per-thread timers, or perf_event_open with a per-thread file descriptor, remove the ambiguity.

The handler runs in a hostile context

A signal handler is not an ordinary function. It executes on the interrupted thread’s stack, with the interrupted thread’s registers saved underneath, and it can arrive at any instruction boundary, including in the middle of a memory allocation or while the thread holds a lock. The set of operations permitted in that context is the async-signal-safe list in signal(7): write, read on an already-open descriptor, _exit, sigaction, a handful of others. malloc is not on it. Neither is printf, and neither is anything that takes a lock the interrupted thread might already hold.

That constraint shapes the design. A profiler’s handler cannot format a stack trace, cannot allocate a node for a call tree, and cannot look up a symbol. It can copy raw addresses into a preallocated buffer and return. All interpretation happens later, in the process that reads the buffer, where locks and allocation are legal.

A second constraint is re-entrancy of the measurement itself. While a handler runs, the same signal is blocked by default, so a nested sample cannot occur on the same thread. But the handler’s own execution is charged to the thread, which means it inflates the very CPU time the timer measures. At a hundred samples per second with a handler that costs tens of microseconds, the handler’s own work is a fraction of a percent of runtime, but it is counted in the samples, so a naive profiler attributes part of its own overhead to whatever function the handler interrupted.

#define RING 4096
static volatile unsigned head;
static unsigned long long pc_ring[RING];
static unsigned long long fp_ring[RING];

static void on_prof(int sig, siginfo_t *si, void *ctx)
{
    ucontext_t *uc = ctx;
    unsigned idx = head % RING;
    /* Both fields come from the interrupted context, not from the handler. */
    pc_ring[idx] = (unsigned long long)uc->uc_mcontext.gregs[REG_RIP];
    fp_ring[idx] = (unsigned long long)uc->uc_mcontext.gregs[REG_RBP];
    head = head + 1;                 /* single writer: no lock needed */
}

The ucontext_t argument is the only place the interrupted register state is visible, and the register names are architecture-specific. The code above is x86-64 Linux; on other architectures the same fields exist under different names. A profiler that reads the frame pointer from inside the handler instead of from the saved context records its own frame and reports a stack that belongs to the profiler.

What a sample actually records

The saved instruction pointer identifies one instruction. Everything else in the profile is derived: the call stack comes from walking the frame pointer chain or consulting unwind tables, the function name from a symbol table, the source line from debug information. Each of those derivations can fail independently, and each failure is silent unless the profiler counts it.

The stack walk itself is the fragile step. Frame-pointer chains require that every function in the path maintain a frame pointer, which optimizers at typical optimization levels are entitled not to do. Table-driven unwinding requires the debug or unwind tables to be present at runtime, which is often false for stripped production binaries. A profiler that walks a chain it cannot trust produces stacks that terminate early or, worse, follow a stale value into unrelated memory and produce a plausible but wrong caller. The mechanics of both approaches, and the failure signature of each, are the subject of Frame Pointers, DWARF Call Frame Information, and What Breaks When Both Are Absent.

What the sample does not record is equally important. ITIMER_PROF and CPU-time timers account for time the thread was running. A thread blocked in read on a socket contributes nothing to the profile no matter how long it waits, so a CPU profile of a service that spends most of its wall time waiting for a downstream dependency shows a small, tidy, entirely misleading picture. That gap is not a defect of the sampling method; it is a property of the clock the sampler chose, and closing it requires a different clock rather than a higher rate.

Frequency, aliasing, and the cost curve

Sampling frequency sets both the resolution and the price. Overhead is the product of the rate and the cost of one sample, so at 100 Hz with a handler and stack copy costing roughly 20 microseconds, the expected overhead is $100 \times 20,\mu s = 2,ms$ per second, or 0.2 percent. The same profiler at 1000 Hz costs ten times that. The arithmetic is the whole trade: fidelity improves with the square root of sample count, cost improves linearly with the rate, so the marginal sample buys progressively less.

The rate also interacts with the workload’s own periodicity. A sampler ticking at exactly 100 Hz against a service whose main loop wakes every 10 milliseconds will interrupt the same phase of that loop every time. The resulting profile is not a sample of the loop’s work; it is a sample of one point in it, repeated. Nothing in the data indicates that this happened, because every sample looks individually valid. The standard mitigations are to choose a rate that is not a divisor of any obvious period in the system and to add jitter to the interval, so that the phase drifts.

There is a subtler aliasing problem at the top end. Hardware-counter sampling can be configured to overflow every $N$ events rather than on a timer, which gives a uniform sample per unit of work rather than per unit of time. That is often the better estimator, because it does not over-weight slow instructions, but the period is still a fixed number and the same phase-locking argument applies to loops whose iteration count is a multiple of it.

The boundary between sampler and interpreter

Every design in this space separates two phases that run at different times and in different privilege. The capture phase is tiny, allocation-free, and bounded in time because it runs on a thread that has been stopped mid-work. The interpretation phase, which resolves addresses to functions, folds stacks into a tree, and attributes cost, runs later and elsewhere.

Keeping that boundary sharp is what makes continuous profiling affordable: the capture path costs a few microseconds per sample, and everything expensive happens once per batch on a machine that is not serving requests. A profiler that symbolises inside the handler, or that allocates while the target is stopped, converts a fixed per-sample cost into an unbounded one and turns a 0.2 percent tax into something that shows up in latency graphs.