How Many Samples a Profile Needs Before Its Numbers Mean Anything

A profile printed by a sampling profiler looks like a table of facts: 41% in the parser, 12% in the allocator. It is not a table of facts. It is a table of estimates, each derived from a few thousand coin flips, and the width of the error bar depends entirely on how many flips were taken. Treating a 200-sample profile as if it were a measurement is the most common way to draw a confident wrong conclusion from a profiler.

Each sample is one draw from the execution’s time distribution

The model is simple. The program’s execution over an interval has some true distribution of time across code locations. The profiler draws N independent samples from that distribution and counts how many landed in each location. For a single location whose true share is p, the count is binomial: mean Np, variance Np(1-p). The estimate p̂ = n/N has standard error

$$\operatorname{SE}(\hat p) = \sqrt{\frac{\hat p,(1-\hat p)}{N}}$$

The independence assumption is where reality leaks in. Samples taken at fixed intervals are not independent draws: code has temporal locality, so consecutive samples correlate strongly, and the effective sample size is smaller than N. A tight loop that occupies 30% of the run will be sampled in long runs rather than scattered uniformly. The variance of the estimate under correlation exceeds the binomial formula, sometimes by a large factor, so the intervals below are optimistic. They are still the right order of magnitude, and they still tell you that small N cannot support fine distinctions.

A 95% interval is roughly p̂ ± 1.96 SE. At N = 100 and p̂ = 0.05, SE is √(0.05 · 0.95 / 100) ≈ 0.0218, so the interval runs from about 0.7% to 9.3%, the estimate is compatible with the function being nearly irrelevant and with it being a tenth of the run. At N = 10,000 the same 5% estimate has SE ≈ 0.0022, and the interval narrows to roughly 4.6% to 5.4%. Ten thousand samples buy half a percentage point of resolution on a small share; a hundred samples buy nothing usable.

The practical translation: to resolve a share of about 1% to within ±10% relative (that is, to distinguish 0.9% from 1.1%), the standard error must be near 0.0005, and since N ≈ p(1−p)/SE², that is on the order of 40,000 samples. That is why profiles of short runs are useless for anything but the largest bars.

Sample rate times duration is the whole budget

The only lever is N = rate × duration × threads-sampled. At 99 Hz over a ten-second run of a single thread, N is about 990. At the same rate over ten minutes it is about 59,000. Overhead scales with rate, not with duration, so extending a run is nearly free in profiler cost while raising the rate is not. This asymmetry is the single most useful fact about sampling profiles: long runs at a modest rate beat short runs at a high rate, for every question that can wait.

# 99 Hz for 60 s on a whole process tree; about 5,900 samples of one busy thread
perf record -F 99 -g -- sleep 60

# The same total sample count spread over a long run costs nothing extra in overhead
perf record -F 99 -g -- ./service --duration 600

The exception is a workload that is not stationary: a service whose behaviour changes over the run (warmup, cache growth, periodic compaction) mixes several distributions, and the aggregate profile describes none of them. There the answer is repeated short windows, each with its own error bars, not one long run.

Counts are not time

The second source of error is aggregation, not sampling. A profile stores, per stack, a count of samples. If the sampler fires at a fixed wall-clock rate, one sample represents one fixed quantum of time and counts are proportional to time. If the sampler fires on a hardware counter, every million cycles retired, every thousand cache misses, then one sample represents a variable amount of time, and a stack with many samples may be one that executes many instructions rather than one that is slow. Mixing the two kinds of sample in one table produces numbers with no unit at all.

Weighting matters in the other direction too. Samples taken during a period when the CPU was running at a reduced frequency represent more wall-clock time per cycle than samples taken at full boost. A profile weighted by cycles will understate the wall-clock cost of code that runs while the core is throttled, which is a real effect on laptops and on thermally limited servers. This is one reason a cycle-based profile and a wall-clock profile of the same program can disagree by tens of percent.

Minimum detectable difference between two profiles

Comparing a profile before a change with one after is a difference of two binomial estimates. The standard error of the difference is the quadrature sum of the two standard errors, so the detectable effect shrinks only as the square root of the sample count. With two runs of 5,000 samples each, a share near 10% has SE ≈ 0.0042 per run, difference SE ≈ 0.006, and the smallest difference distinguishable at 95% is about 1.2 percentage points. A change that moves a function from 10% to 9% is invisible at that budget; a change from 10% to 7% is not.

Two further cautions apply when scanning a profile for “the” difference. First, the comparison is made across hundreds of stacks simultaneously; at a 95% threshold, roughly one in twenty comparisons of unchanged code will cross the line by chance. Selecting the largest apparent change out of many and reporting it as significant is close to guaranteed to find noise. Second, the sampling error is not the only error: address randomisation, different inlining decisions from a rebuild, and different allocator layout all shift stacks between runs, so two profiles of nominally identical code differ by more than the binomial model predicts.

The honest procedure is to fix a threshold from the sample count before looking at the data, “only differences above two standard errors are considered”, and to confirm a suspected regression with a second pair of runs. The mechanics of doing the comparison at scale, including how to align stacks whose frames moved, are covered in differential profiles.

Where the model breaks down entirely

Several situations make the binomial arithmetic inapplicable rather than merely optimistic.

Short-lived threads. A thread that lives for less than a sampling period has a probability of being sampled proportional to its lifetime, and the profiler sees it rarely or never. Its work is not underrepresented by a few percent; it is missing. A profile of a request-per-thread server can show almost no time in the request handler while the handler is where all the work happens.

Blocked time. A CPU-time sampler accrues samples only while the thread is on-CPU. Time spent waiting on a lock, a socket, or a page fault contributes nothing, so the profile’s total is not the program’s elapsed time and percentages are of CPU time, not of latency. The missing half of the picture is the subject of off-CPU profiling.

Skid and attribution error. A counter overflow is detected some instructions after the event that caused it; the recorded instruction pointer can therefore sit past the instruction of interest, in a neighbouring function, or in a loop iteration that did not cause the miss. This is worst for events with long latencies, and it is why precise-event mechanisms exist.

Unresolvable frames. Samples are recorded as addresses. If symbolisation fails (stripped binaries, inlined code, generated code without a symbol map), those samples are attributed to an unknown bucket. A large “unknown” fraction does not merely reduce precision; it can hide the very code that was being investigated. The mechanisms involved are in symbol attribution.

Non-stationarity within the run. If the workload changes character mid-run, the aggregate is a mixture and its error bars describe the mixture, not either phase. Splitting by time window is the only fix, and it requires the samples to carry timestamps, which most formats do.

Choosing a run length from the question

The useful discipline is to state the question in terms of a share and a tolerance, then compute the sample budget backwards. To establish that a function is a major cost, say, more than a quarter of the time, a few hundred samples suffice, because the interval at 25% is narrow in absolute terms. To rank two functions that differ by three percentage points, several thousand samples per run are needed. To decide whether a 0.5% function is worth optimising at all, no sampling rate will answer it; the correct instrument is a targeted measurement of that function alone.

None of this makes sampling profiles weak. It makes them cheap: a few thousand interrupts produce a picture of an entire program at a cost the program barely notices. The weakness is in reading them as exact. A profile is a survey with a stated sample size, and the sample size belongs next to every number taken from it.