How Sampling Profilers Turn Timer Interrupts into Execution Profiles

A sampling profiler never sees the whole program. It interrupts execution at fixed points, records where the instruction pointer happened to be, and treats the resulting multiset of stacks as an estimate of how the program spent its time. Everything about the technique, its low overhead, its blind spots, its statistical error, follows from that substitution of a sample for a census.

The measurement is an interrupt

The simplest implementation is a process-wide interval timer. setitimer(ITIMER_PROF, ...) decrements a counter while the process is running, and the kernel delivers SIGPROF when it expires. ITIMER_PROF counts user plus system CPU time for the whole process; ITIMER_REAL counts wall-clock time and delivers SIGALRM instead. On expiry the kernel builds a signal frame on whatever thread is currently executing and returns to userspace at the handler entry point. The handler can read the interrupted context, on Linux the ucontext_t passed to a SA_SIGINFO handler exposes uc_mcontext, and that saved register set contains the instruction pointer and stack pointer at the moment of the interrupt.

#define _GNU_SOURCE        /* REG_RIP and the ucontext register names */
#include <signal.h>
#include <string.h>
#include <sys/time.h>
#include <ucontext.h>

static void on_prof(int sig, siginfo_t *si, void *uc)
{
    ucontext_t *ctx = uc;
    /* PC at the moment of interruption; the leaf of the sampled stack */
    void *pc = (void *)ctx->uc_mcontext.gregs[REG_RIP];
    record_sample(pc, ctx->uc_mcontext.gregs[REG_RSP]);
}

int main(void)
{
    struct sigaction sa;
    memset(&sa, 0, sizeof sa);
    sa.sa_sigaction = on_prof;
    sa.sa_flags = SA_SIGINFO | SA_RESTART;
    sigemptyset(&sa.sa_mask);
    sigaction(SIGPROF, &sa, NULL);

    struct itimerval t = {
        .it_interval = { 0, 10000 },   /* 100 Hz */
        .it_value    = { 0, 10000 },
    };
    setitimer(ITIMER_PROF, &t, NULL);
    run_workload();
}

That sketch is the whole idea, and it is also where the trouble starts. The interval is expressed in CPU time, so a process blocked in a syscall stops accumulating ticks and the profiler silently reports zero samples for the blocked interval. SIGPROF is a standard signal, not a real-time signal, so it is not queued: if a second expiry occurs while the first handler is still running or the signal is blocked, the second is merged into the pending one. Under a heavy handler the effective rate collapses toward the handler’s own duration, and the profiler under-reports exactly the busiest code.

Modern Linux profilers avoid signals almost entirely. perf_event_open(2) creates an event that the kernel programs onto a hardware or software counter, and on overflow the kernel writes a sample record into a ring buffer that userspace has mapped with mmap. There is no signal frame and no per-sample handler: the kernel writes the record and, once a configured wakeup threshold is crossed, wakes a reader blocked on the event’s file descriptor. The record contains the timestamp, thread and CPU identifiers, the sampled instruction pointer, and, if call-chain capture is enabled, the stack.

Why sampling beats instrumenting every call

The competing technique inserts a call at every function entry, -pg with mcount, or the kernel’s fentry hooks, and accumulates exact call counts. Its cost is proportional to the number of calls, which is the wrong quantity. A leaf function invoked a billion times pays a billion hook invocations, while a slow function invoked once pays one, and the profiler’s overhead lands hardest on the code that is already efficient. Sampling inverts this: cost is proportional to elapsed time and sample rate, independent of how many times anything is called.

Instrumentation also changes the program it measures. A call inserted at entry can prevent inlining, alter register allocation, and shift code layout across cache lines. Sampling leaves the instruction stream untouched; the only perturbation is the interrupt itself, which flushes pipelines and pollutes caches at each tick.

Frequency, aliasing, and the 99 Hz convention

The sample rate sets the resolution of the result. At 100 Hz a one-second window yields about a hundred samples, enough to see that a function consumes roughly a third of the time, not enough to distinguish 3% from 5%. Rates in the range of tens to a few thousand hertz are the practical band: below that the noise dominates, above it the interrupt cost and cache disturbance become part of what is being measured.

There is a second constraint that is easy to miss. If the sampling period shares a common factor with a periodic activity in the program (a 10 ms polling loop, a 4 ms frame deadline, the kernel’s own scheduling tick), then successive samples land at the same phase of that activity and the profile becomes a consistent lie. A workload that wakes every 10 ms and spends 1 ms working will look either idle or saturated depending on where the sampler’s phase sits. Choosing a rate that is coprime with the expected periodicities decorrelates the phase. This is the origin of the widespread habit of running at 99 Hz rather than 100: the odd number cannot divide, or be divided by, the common tick rates of 100, 250, and 1000 Hz that kernels use for HZ.

The kernel’s tick matters for a related reason. A CPU-time timer only advances while the thread is on-CPU, so sample timestamps quantise to scheduling decisions. Wall-clock sampling at a fixed rate over a thread that is frequently preempted produces samples clustered around the moments it regains the CPU, which over-represents code that runs immediately after a wakeup.

What a sample record contains, and what it costs

A minimal record is a timestamp and an instruction pointer. In the perf ring buffer the sample header is a few tens of bytes, and an eight-frame call chain adds 64 bytes of addresses on a 64-bit machine; a thousand samples per second therefore moves on the order of a hundred kilobytes per second per thread, which is why the ring buffer is mapped rather than read through a syscall per sample.

The expensive part is not the record but the stack walk. Reading a frame-pointer chain costs one dependent load per frame, and each load can miss cache; unwinding through DWARF call-frame information costs far more, sometimes microseconds per sample, which is why high-rate profilers prefer frame pointers or hardware-assisted branch stacks. The trade-off is direct: higher rate means finer resolution and proportionally more overhead, and the overhead is not uniform, it is concentrated in the code that happens to be interrupted.

A sample only exists if the profiler gets control. Threads that live for less than one sampling period may never be observed at all, so a program that creates a short-lived thread per request can show a profile dominated by the long-running threads and none of the work actually done per request. Threads that block in a syscall accumulate no CPU-time ticks, so a program spending most of its life waiting on a lock or a socket appears fast and idle; measuring that time requires a different mechanism, covered in off-CPU profiling.

Attribution is the other blind spot. The instruction pointer identifies where execution was, but the sample is charged to the leaf frame, and the leaf may be a memory copy inside a library, an allocator slow path, or a spin loop whose caller is the interesting part. Symbolisation failures compound this: inlined frames are absent from the stack, dynamically generated code has no symbols, and a sampled address in the middle of an instruction cannot be mapped back to a source line without the debug information described in symbol attribution.

Reading the counts as statistics

Each sample is a Bernoulli trial for the question “was execution in region R”. With n samples landing in R out of N total, the estimated fraction is p̂ = n/N, and the sampling error shrinks only as the square root of the count: a hundred samples put a band of roughly ±4 percentage points around a 5% estimate, while ten thousand samples narrow it to about ±0.4 points. The arithmetic of that interval, and the sample budget implied by a question, are worked through in sampling error and confidence. What matters here is the consequence: the width of a bar in a flame graph, discussed in constructing flame graphs, is a count with an error bar attached, and differences smaller than that error bar are not signal.

The sampling rate also determines which questions the data can answer. At 100 Hz, the expected number of samples inside a 100 µs function is 0.01, so such functions are effectively invisible individually; they appear only in aggregate under their callers. Raising the rate to resolve them costs overhead in proportion, and no rate resolves code that runs for less than one period. For sub-period work the answer is instrumentation or tracing, not a faster timer.