Merging Profiles Across a Fleet Without Losing the Tail

A profile from one host describes where that host’s CPU time went. A profile of a fleet describes where the fleet’s CPU time went, and the difference between those two sentences is the difference between a sum and a weighted sum. Getting the weight wrong produces a graph that is arithmetically correct and operationally useless, because it describes the resource consumption of the fleet rather than the cost of the code.

The distinction has a concrete form. Suppose a service runs on two host types: many small instances handling most requests, and a few large instances handling a smaller share but running a different code path for legacy clients. Summing raw sample counts weights each host by how much CPU it happened to use, so the legacy path’s share of the merged profile is its share of total CPU, which is what a capacity planner wants. Normalizing each host to its own total weights each host equally, so the legacy path’s share is its share of that path’s hosts’ work, which is what an engineer optimizing the legacy path wants. Neither is wrong; they answer different questions, and a pipeline that silently picks one produces numbers that get misread.

What a sample count means before normalization

Each sample represents a fixed quantum of the clock the profiler chose, typically a CPU-time interval or a wall-clock interval. Summing samples across hosts therefore sums time, provided the hosts used the same clock at the same rate. If one host samples at 100 Hz and another at 200 Hz, summing raw counts weights the second host twice as heavily as the first for the same amount of work. Rate normalization is the first and least controversial correction, and it is easy to forget when hosts are configured by different deployment templates.

The second correction is clock choice. A CPU-time profile of an idle host yields few samples, because the host was not running. A wall-clock profile of the same host yields many samples, all attributed to whatever the idle loop was doing. Merging the two produces a mixture in which idle time from one host dilutes real work from another. The clock must be uniform across the fleet, or the merge must be restricted to hosts using the same clock.

The third is sample rate relative to core count. A host with sixty-four cores sampling at 100 Hz per core produces 6400 samples per second, while a host with four cores produces 400. A per-core rate is the natural configuration for a CPU profiler, but it means raw counts scale with core count, so a fleet merge weights large hosts proportionally. That is defensible for capacity questions and misleading for code questions, where a function that runs only on the large hosts will appear to dominate.

Three normalizations and what each preserves

Normalizing by total samples makes each host contribute an equal share to the merged profile. The merged percentages then read as “of the work done across hosts, weighted equally”, which is the right frame for comparing code paths that are deployed unevenly. It is the wrong frame for capacity, because a host with a hundred times the traffic contributes no more than a host with none.

Normalizing by request count, or by whatever unit of work the service processes, converts the profile into cost per unit. This is the most useful normalization for engineering questions, since it makes profiles comparable across hosts with different traffic and across time periods with different load. It requires the work unit to be measurable and to be attributed to the samples, which usually means correlating the profile with a counter rather than deriving it from the profile itself.

Not normalizing at all preserves total resource consumption and answers capacity questions directly. It is also the easiest to implement, which is why it is the default in most pipelines, and it is the one that most often gets misread as a code-quality signal.

A practical arrangement keeps the raw counts and records the normalizers alongside them, so a query can divide by whatever denominator the question needs. Storing only normalized values forecloses the other questions permanently.

Why the tail behaves differently from the mean

Aggregating CPU time is additive, so a merged profile’s percentages are means over the fleet. A tail question is not additive in the same way. The slowest requests in a fleet are not the slowest requests on each host averaged; they are the slowest requests overall, and they may come disproportionately from one host, one build, or one code path.

A merged CPU profile cannot answer that question, because it has thrown away the per-sample timing. What it can do is show which code paths are disproportionately represented when the slow cohort is filtered, provided the filter is applied before aggregation. If the pipeline aggregates first and filters later, the tail’s code path is averaged into the whole and disappears.

This is the operational argument for keeping the dimensions that a filter might need: host, build identifier, deployment version, and whatever label identifies the cohort. Cardinality is the constraint, and the usual compromise is to keep a small set of dimensions at full fidelity and to pre-aggregate the rest. The trade is the same one that appears in metrics, where the choice of labels determines which questions are answerable later.

There is a second tail effect specific to profiles. A profile is a sample of time, and a rare but expensive code path contributes few samples because it runs rarely, even if it accounts for most of the latency of the requests that hit it. The sample count reflects CPU consumed, not latency imposed. A path that blocks for a second and computes for a millisecond appears in a CPU profile as a millisecond. Attributing the other second requires the off-CPU side described in Off-CPU Time and Scheduler Delay as a Profile, and merging that data has the same normalization problems with an additional one: waiting time is not conserved across hosts in the way CPU time is.

Alignment: builds, symbols, and time

Merging requires that the same stack means the same thing on every host. Three things break that.

Build divergence is the first. If half the fleet runs one build and half another, the same address symbolizes to different functions, and the same function occupies different addresses. Symbolizing before merging resolves the second problem and leaves the first, which is legitimate: the two builds really do have different code, and a merged profile of both is a profile of two programs. The fix is to merge per build and compare across builds, not to merge across builds and hope.

Inlining decisions are the second. The same source function may be inlined at one call site and out-of-line at another, and the choice can differ between builds compiled with different flags. A merged profile then shows the function at different depths on different hosts, and the tree structure no longer aligns. This is not an error, but it makes node-by-node comparison between builds unreliable, and it argues for comparing at the level of the function rather than the call path when builds differ.

Time alignment is the third and is often overlooked. Merging profiles from different collection windows compares different workloads. A deploy that changes traffic distribution between two windows produces a profile difference that has nothing to do with the code change. The mitigation is to merge over matched windows, at matched points in the traffic cycle, and to record the window boundaries with the merged result so that a later reader knows what was compared.

Cardinality and the cost of keeping everything

The number of distinct stacks in a fleet profile is the product of code diversity and data diversity, and it is much larger than any single host’s. A merged profile of a thousand hosts running identical builds still has the union of their data-dependent call paths, which for a parser or an interpreter can be enormous.

Storage grows with distinct stacks, not with samples, so the compression that makes per-host profiles cheap stops working at fleet scale. The usual responses are to cap depth, to prune nodes below a threshold, and to keep a small number of exemplar stacks for any node that exceeds a threshold. Exemplars are the useful compromise: they preserve enough per-sample detail to answer “what did a stack in this node look like” while bounding the stored cardinality.

Whatever pruning is applied must be applied after the merge, not before. Pruning per host removes nodes that are rare on each host but common across the fleet, which is exactly the long-tail code path that a fleet-wide view is supposed to reveal. A threshold applied per host at a thousand hosts will delete every path that occurs on a tenth of them, and the merged profile will show a fleet that does nothing but serve the common case.

Comparing two merged profiles

The comparison that operators actually want is between two merged profiles, before and after a change. Doing it correctly requires the same normalizer on both sides, the same build on both sides or an explicit acknowledgement that they differ, and a statistic that accounts for sampling error. A difference of one percent between two profiles of ten thousand samples each is within the noise described in Sampling Error, Skid, and How Many Samples a Profile Needs, and reporting it as a regression is a way to generate work that does not exist.

The useful comparison is structural rather than numeric: which call paths appeared, which disappeared, and which changed depth. That comparison is robust to sampling noise in a way that percentage deltas are not, and it catches the changes that matter, such as a function moving from a shared library into the binary or an inlining decision flipping.