The Overhead Budget of a Sampling Profiler

Continuous profiling in a serving system is a budget problem before it is a tooling problem. The profiler runs alongside the workload on the same cores, consumes the same memory bandwidth, and shares the same caches. Every microsecond it spends collecting is a microsecond the workload does not have, and the fraction is set by a design decision that is usually made once and never revisited.

Stating the budget as a single number is misleading, because the cost is not one thing. It is the per-sample capture cost multiplied by the sample rate, plus the cost of buffering and transferring the raw data, plus the cost of aggregation wherever that happens, plus the cost of the artifacts the profiler keeps resident. Each term scales differently with the parameters, and the largest term changes with the deployment shape.

Capture cost is a per-sample constant

The capture path is the part that runs on the sampled thread. On a signal-based profiler it comprises the kernel’s signal delivery, the handler’s entry, the register save, the stack walk, and the copy of addresses into a buffer. The stack walk dominates: a frame-pointer walk of thirty frames is roughly sixty dependent loads, and a DWARF walk is thirty table searches with a dependent load at each frame.

The arithmetic that matters is that this cost is paid once per sample, so total overhead is linear in the sample rate:

$$ \text{overhead fraction} = f \times c $$

where $f$ is samples per second and $c$ is the cost per sample in seconds. At 100 Hz and 20 microseconds per sample, the fraction is $100 \times 20 \times 10^{-6} = 0.002$, or 0.2 percent. At 1000 Hz it is 2 percent. Those figures are approximate and depend on stack depth and cache state, but the linear relationship is exact and it is the one to design against.

The frame-pointer case deserves emphasis because it is cheap enough to change the calculus. A frame-pointer walk has no metadata to fetch, so its cost is a few cache lines of stack, which on a hot thread are already resident. A DWARF walk touches .eh_frame, an index, and often the debug file’s memory, none of which the workload was using, so it evicts workload data from cache in addition to its own instruction cost. The second-order effect can exceed the first: a profiler that adds 0.2 percent of CPU time but evicts a working set from L2 can cost several percent of throughput.

Buffering and the transfer term

Samples must leave the captured thread without blocking it. The standard structure is a per-thread ring buffer written lock-free by the sampler and drained by a separate collector. The writer’s cost is a store and an index increment; the reader’s cost is a memcpy out of the ring plus whatever serialization the transport requires.

Ring size is the parameter that converts a smooth cost into a bursty one. A ring of 4096 entries at 100 Hz covers about forty seconds of samples, which absorbs a collector stall. A ring that fills drops samples, and a profiler that drops samples silently reports a profile of the periods when the collector was healthy, which is a biased subset.

The transfer term is where continuous profiling differs most from on-demand profiling. A one-minute capture can afford to write raw stacks to disk and analyse them afterwards. A continuous collector running on every host produces a stream, and the stream’s volume is proportional to sample rate times average stack depth times bytes per address. At 100 Hz, depth 30, and 8 bytes per address, one thread produces 24 kilobytes per second of raw addresses before compression. A process with sixty-four threads produces 1.5 megabytes per second, and a host with several such processes produces enough that shipping raw data is a bandwidth decision rather than a detail.

That number is the argument for aggregation close to the source. Folding stacks into a call tree before transmission replaces per-sample records with per-node counts, and the count of distinct nodes in a real program is far smaller than the count of samples. The compression is not lossless in the sense that it discards sample timing, but it preserves everything a call-tree profile needs. The structure of that folding is treated in Folded Stacks, Call Trees, and What Aggregation Destroys.

Where aggregation happens decides who pays

Three placements are possible, and they differ in which resource is consumed.

Aggregation on the sampled host uses that host’s CPU, competing directly with the workload. Its advantage is that raw stacks never leave the machine, so the network and storage terms vanish. Its cost is that the folding work is proportional to sample count, and folding is not free: hashing a thirty-frame stack and walking a tree to increment a node costs on the order of a few microseconds, comparable to the capture itself. Doing it inline in the handler is impossible, so it happens on the collector thread, which means it consumes a core.

Aggregation in a central service moves the cost off the host entirely and puts it on machines sized for the purpose. The price is bandwidth for raw stacks and a loss of locality: the central service must have the symbolization artifacts for every build on every host, which is the retention problem described in Build IDs, Separate Debug Files, and Symbol Resolution at Scale.

Aggregation in the kernel, using in-kernel maps that accumulate counts keyed by stack, eliminates the userspace transfer entirely. The kernel path costs a hash lookup and an increment per sample, with no ring buffer and no collector thread. It requires a kernel facility capable of holding a map keyed by a stack trace, and its limits are the map’s entry count and the cost of hashing a long key. This is the design point behind kernel-side profilers, discussed in In-Kernel Aggregation and the Cost of Getting Stacks Out.

The trade across all three is the same shape: move the work to where CPU is cheapest, and pay for it in data movement or in operational coupling.

The resident cost that does not scale with rate

Some overhead is independent of sample rate and is therefore easy to overlook. The profiler’s own memory footprint is one: a symbol cache holding resolved ranges for every loaded object, a call-tree structure holding a node per distinct stack, and the decompressed debug information for each build. A tree node is on the order of a hundred bytes; a program with a hundred thousand distinct stacks has ten megabytes of tree, which is tolerable, while a program generating millions of distinct stacks through deep recursion and varying call paths has hundreds of megabytes, which is not.

Distinct stack count is the variable that breaks naive designs, and it is driven by data-dependent call paths rather than by code size. Recursive parsers, interpreter dispatch, and generic code instantiated many ways all produce combinatorial stack diversity. The standard mitigations are to cap stack depth (dropping frames beyond a limit), to collapse recursion into a repeated-frame marker, and to prune nodes below a count threshold at collection time. Each is a lossy choice, and each should be visible in the data rather than applied silently.

There is also a fixed per-sample cost in the kernel that does not appear in the handler: programming the counter, arming the overflow, and delivering the interrupt. Hardware-counter sampling pays this per event; a timer pays it per tick. The costs are small individually and are counted in the same linear budget.

Measuring the overhead instead of assuming it

The only honest way to state overhead for a particular system is to measure it, and the measurement has a specific shape. Comparing throughput with the profiler on against the profiler off is the direct approach, but it conflates the profiler’s CPU consumption with its cache and memory effects, and it requires a stable workload.

A cheaper proxy is to read the profiler’s own consumption from /proc, which reports per-thread user and system time in clock ticks and the process’s resident set size in pages. Summing the collector threads’ time and dividing by elapsed wall time gives the CPU fraction directly, and the resident set gives the memory term. Neither captures cache interference, and neither captures time stolen from the workload by the interrupt itself, but both are available without perturbing the workload and both are stable enough to alert on.

The budget itself should be stated as a target before the tool is chosen. A common shape is a fixed CPU fraction per host, with the sample rate derived from it rather than chosen first: if the budget is 1 percent and a sample costs 20 microseconds, the affordable rate is 500 Hz across the host, divided among threads. Deriving the rate from the budget makes the trade explicit and prevents the situation where a rate chosen for a single-threaded prototype is applied unchanged to a host running sixty-four threads, where it costs sixty-four times as much.

The asymmetry that makes profiling worth the tax

The final consideration is that overhead is paid continuously and the benefit arrives rarely. A profiler that costs 1 percent of a fleet’s CPU is spending that continuously for a benefit that materializes when a regression appears or when a capacity decision needs a number. The trade is only favourable if the data is retained long enough and complete enough to answer those questions when they arise, which is why retention and coverage matter more than the last factor of two in sample rate. A cheap profiler with sparse coverage answers nothing; a slightly more expensive one that covers every host and keeps months of history answers questions that no amount of ad-hoc measurement can.