Comparing Two Profiles to Find a Regression That Is Real

Two profiles of the same program differ. The question is whether the difference comes from the change that was made or from the sampling, the machine, the input, and the compiler’s mood. A differential profile is the tool for answering it, and the arithmetic behind it is more subtle than subtracting two percentages, because a program whose total time changed has changed every other number along with it.

Two ways to normalise, and only one of them shows a regression

Take a profile before a change with A total samples and one after with B. The first temptation is to compare absolute counts: a function with 4,000 samples before and 4,500 after has grown. That comparison answers “did this function consume more CPU time”, which is a real question, but it is contaminated by whatever else happened. If the whole program got 20% slower, every function gains samples, and nothing appears anomalous. Absolute comparison finds global regressions well and local ones badly.

The second option is to normalise each profile to its own total before differencing, so that every node is a share of the run. This is what a differential flame graph does: both tries are rescaled to a common total, and each node’s value is the change in share. Red means the node’s share of the new run exceeds its share of the old one. The catch is that when everything slows down by the same factor, no share changes and the graph is uniformly grey, even though the program is measurably slower. Normalised comparison finds local regressions well and global ones badly.

Neither is sufficient alone, so the total sample counts belong next to the graph. A run whose total grew by 20% while its shares stayed flat is a whole-program regression, and the search for it has to look at counters rather than at the profile’s shape.

Building the union trie

The layout requirement is that both profiles be drawn on the same set of nodes. Inserting each try separately and rendering them overlaid produces a graph where a node present in only one profile shifts everything to its right, so the eye compares a red bar with a blue bar that represents a different subtree. The fix is a union trie: fold both folded-stack files into a single structure whose nodes are the union of both key sets, with a zero count where a node is absent from one side.

def diff_trie(a, b, node=None):
    """Merge two folded-stack tries into (name, before, after) nodes."""
    node = {} if node is None else node
    names = set(a) | set(b)
    for name in sorted(names):                    # stable order keeps layouts comparable
        child_a, child_b = a.get(name, {}), b.get(name, {})
        node[name] = diff_trie(child_a, child_b)
    node[""] = (a.get("", 0), b.get("", 0))       # self counts for this frame
    return node

Zero counts are not merely a rendering convenience. They make the arithmetic of the difference well defined for every node: a function that exists only in the new build is a node with before-count zero and a large positive delta, which is exactly the shape a newly introduced hot path takes. Treating missing and zero as different cases is the most common source of wrong conclusions in a differential view, because an absence and a zero are indistinguishable in the data.

The noise floor comes first

A difference is only a regression if it exceeds what two runs of identical code would produce. That floor is computable before any change is made, and it should be computed, because it is per-node and varies by an order of magnitude across the graph.

The standard error of a share estimate is sqrt(p(1-p)/N) per run, so the difference of two shares has standard error sqrt(p_a(1-p_a)/N_a + p_b(1-p_b)/N_b). At 20,000 samples per run, a node with a 5% share has a difference standard error of about 0.0022, so a change smaller than roughly 0.4 percentage points is within two standard errors and should be ignored. The same computation at 2,000 samples per run gives roughly 1.4 points, which is large enough to swallow most real regressions in small functions.

# Two runs of the same code, folded and compared, to measure the noise floor
perf record -F 99 -g -o before.data -- ./service --workload w1
perf record -F 99 -g -o after.data  -- ./service --workload w1
perf script -i before.data | stackcollapse-perf.pl > before.folded
perf script -i after.data  | stackcollapse-perf.pl > after.folded

Two subtleties make the computed floor optimistic. Samples within a run are not independent, as sampling error explains, so the effective N is smaller than the nominal one and the true floor is higher. And the runs differ in ways unrelated to the change: address randomisation, allocator layout, filesystem state, and the identity of the cores the threads were placed on. Empirically measuring the floor with a pair of control runs costs one extra measurement and removes the need to reason about any of these.

Frame identity is the hard part

The differencing arithmetic assumes that a node name in the old profile refers to the same code as the node name in the new profile. That assumption fails often, and each failure produces a graph that looks like a regression and is an artefact.

Inlining changes. Adding a small function to a hot path can cause the compiler to inline or stop inlining a neighbouring function. A subtree of frames collapses into its parent or expands out of it, and the parent’s share jumps while the actual instruction stream is nearly unchanged. The frames are right; the tree shape is different. The same build with a different optimisation level is a different tree even with identical source. This is the phenomenon described in symbol attribution in static form, and the practical response is the same: resolve to the deepest available frame, including inline frames, so that the comparison happens at the level that did not move.

Renames and moves. A function that is renamed, or a source file that is moved, changes the node name, and its cost appears as a large negative delta on the old name and a large positive delta on the new one. The two deltas cancel in the total and look like two independent findings. Sorting the collapsed stacks and comparing the name sets before looking at the deltas catches this immediately: a pair of nodes with equal and opposite changes is usually one node with two names.

Static functions and identical names. Symbolisation can produce the same name for two different functions, either because both are static in different translation units or because demangling stripped the namespace. Their merged node is stable across builds only if the two are always merged in the same proportion, which is not guaranteed. Including the module or object in the frame label prevents the merge at the cost of noisier layouts.

Truncated stacks. If the two runs used different unwinding modes, or if one run produced a large number of unresolved frames, the tries have systematically different depth. Widths at the same level then no longer sum to the same total, and comparing a node whose children differ in depth by one between builds attributes real time to the wrong level. Both runs must be collected with identical unwinding settings, which is one reason the collection parameters belong in the profile’s metadata.

Reading the colour, and reading the grey

With both sides normalised to the same total, each node’s delta is drawn with saturation proportional to the magnitude. Two readings are easy and one is a trap.

A wide red node at a shallow level means a whole subsystem increased its share, and its children show where within it. Following the red down the tree finds the deepest node whose children are all grey, which is the code that changed rather than the code that merely contains it. That descent is the productive motion: the parent’s redness is inherited from one of its children, and the leaf-most red node localises the change.

A wide red node whose children are all grey usually means a self-cost increase: the function’s own body got slower, without changing what it calls. In a normalised view this is a genuine regression at exactly that function.

Grey is not a third finding. Grey means the difference was smaller than the threshold, which may be because the node is boring or because the sample count could not resolve it. Nodes that are grey because of low counts are indistinguishable from nodes that are grey because nothing happened, and the honest way to separate them is to display the count alongside the delta, or to filter out nodes below a count floor and state the floor.

Where a differential view stops being enough

Within-run comparison is the mode most often missed. The same profile split into an early window and a late window shows the effect of warming caches, growing heaps, and filling tables. A node that is red in the late window against the early one is a cost that grows with time, and that pattern is invisible in a before/after comparison of two runs, both of which mix the phases. Splitting by time requires timestamps on samples and a decision about window boundaries; one-second windows are the usual default because they are coarse enough to hold a few hundred samples at a normal rate.

Interaction between changes is the other limit. Two independent optimisations may each show a positive delta and together show nothing, because they removed the same cost from two directions; a duplicate computation and a cache that made the duplicate cheap. Differencing is linear and the program is not, so a differential profile explains a change only when the change is small enough that interactions stay second-order. When it is not, the comparison to trust is between two builds of the same source with one feature flag toggled, which isolates the change without changing everything else.

The final caution is about the sample source. Two runs profiled with different event sources are not comparable at all: one measures cycles, the other wall-clock time, and the shares differ systematically in memory-bound code, as discussed in sampling profilers. Fixing the source, the rate, the unwinding mode, the workload, and the machine is a short list of preconditions, and a differential graph whose preconditions are unstated is not evidence of anything.