Choosing a Workload Whose Profile Describes Real Traffic

A profile is always a profile of a particular execution. The same binary, run twice with different input sizes, different resident set sizes, or a different number of concurrent clients, will produce profiles whose hot functions differ not by percentages but by identity. Choosing the run is therefore part of the measurement, not preparation for it, and a profile of an unrepresentative run is not a rough answer; it is a precise answer to a question that was not asked.

The two failure directions

Workload choice fails in two opposite ways, and they are not equally obvious.

A synthetic workload that is too simple concentrates cost in places that real traffic does not reach. A loop that parses the same small request repeatedly fits in the instruction cache, the branch predictors learn it perfectly, and the allocation pattern is identical every iteration, so the allocator’s fast path always hits the same size class. The profile shows a tight, cheap inner loop and almost nothing else. Optimising it produces a measurable improvement on that loop and no change in production, because production traffic never presented that loop with the same input twice.

A synthetic workload that is too variable points the other way. Randomising every parameter, including request size, concurrency, and data distribution, produces a run that spends its time in the slow paths of everything: the allocator’s large-object path, the parser’s error handling, the cache’s eviction code. The profile looks like an accurate survey of all the code and is a survey of a mixture of behaviours that no single real workload exhibits, since the mixture is dominated by whichever extremes the generator happened to emit.

Both failures share a cause: the profile reports the average over the run, and any mixture of behaviours averages to something that may match nothing.

What must match between the run and the target

Four properties of the run change the profile materially, and each can be matched or mismatched independently.

Input distribution. Not the mean size, but the shape. Request bodies that are mostly small with a heavy tail produce a different allocator and cache behaviour than uniformly sized bodies of the same mean. The relevant statistic is often the tail, because the tail is where the slow paths live, and a generator that samples from a log-normal distribution with a well-chosen variance approximates real traffic better than a fixed size does.

Resident set size relative to cache. Data that fits in last-level cache is served from it; data ten times larger causes misses on nearly every access. The difference in cost is roughly an order of magnitude per access, so a run whose working set is too small measures a program that is an order of magnitude faster per element and whose hot functions are the ones that do arithmetic rather than the ones that wait for memory. Scaling the dataset until the profile stops changing is the practical test: when doubling the data no longer shifts the distribution of samples, the working set has reached the regime that matters.

Concurrency. A single-threaded run cannot exhibit lock contention, cache-line bouncing, or allocator contention, and those are frequently the largest single costs in a multithreaded program. A run at too high a concurrency measures a different program again: one dominated by queueing, where the on-CPU profile shows spin loops and the useful work is a minority of samples. The profile should be taken at the concurrency the program is designed to run at, with the client count that keeps it near its operating point rather than saturated or idle.

Call path. A microbenchmark that calls a library function directly omits the layers above it. If the production path reaches the function through three levels of abstraction and a virtual dispatch, those layers may dominate, and the microbenchmark’s profile is of code that the production run spends a fraction of its time in.

Clock, frequency, and the machine

A profile of a program running on a machine whose clock is boosted is a profile of a program with more cycles available, and the shares change because the memory hierarchy does not speed up with the core. A workload that is memory-latency-bound at 2 GHz spends a larger share of its time waiting per instruction than the same workload at 3.5 GHz. Turbo behaviour, thermal throttling, and the governor’s choice of frequency all vary during a run, so a profile whose samples are spread over a long run may average over several frequency regimes.

Frequency invariance is a related trap on servers. If the profiler’s sampling source is the cycle counter, the number of samples per unit of wall-clock time varies with the frequency, so a phase of the run that executes at a lower frequency contributes fewer samples per second and is underweighted relative to its wall-clock duration. A wall-clock sampler avoids this and instead overweights phases where the thread was descheduled. Neither source is neutral, which is why the sampling source belongs in the metadata of any profile that will be compared with another.

Hardware configuration matters as much as frequency. Sibling threads sharing a core, NUMA placement of the allocation, and whether transparent huge pages are enabled all change which samples land in memory management code. The most common mistake of all is comparing a profile from a wide machine with one from a laptop and attributing the difference to the code.

Warmup, steady state, and the phases between them

Almost every program has phases: startup, where code is loaded and caches are cold; warmup, where the allocator grows and the JIT or interpreter compiles hot paths; steady state; and teardown. A single aggregate profile mixes them, and because startup is often slow in absolute terms it can dominate a short run’s samples.

The fix is to segment, and the segmentation must be driven by evidence rather than by a fixed sleep. The evidence is usually in a coarse time series: samples per second, allocation rate, resident set size, or the count of a specific event, sampled once per second. When those series flatten, steady state has been reached. Profiling only after the flattening removes the startup phase, and profiling the startup phase deliberately, as its own run, is a legitimate and often neglected exercise for programs whose latency at launch is a requirement.

Warmup is not only a performance artefact; it changes which code exists. A JIT compiler that has not yet compiled a method is executing an interpreter, and the profile of the pre-compilation phase describes code that will never execute in that form again. Profiling JIT output requires the runtime’s own symbol source, which is the subject of profiling managed runtimes.

Instruments that perturb the run

The measurement apparatus is part of the workload. A profiler that writes to disk, holds a lock around its buffer, or runs a userspace symboliser in the sampled process adds cost that itself depends on the sampling rate. The cost is usually small in aggregate and concentrated in specific code: the signal delivery path for signal-based samplers, the ring-buffer consumer, and the page faults on first touch of the sample buffer. A profile taken at a rate ten times higher than usual may show those paths as new hot functions, and they are the profiler’s own footprint.

Tracing is a sharper version of the same problem. A syscall tracer that stops the process at every entry and exit changes the timing of everything, removes concurrency, and can turn a hot loop into a sequence of synchronous stops. The result is a correct record of the order of operations and an incorrect profile of their cost. The two questions require different instruments, and choosing the wrong one produces a number that is precise and irrelevant.

Verify representativeness before optimising

The cheapest check is a small set of counters compared between the candidate run and the target environment. Retired instructions per request, cycles per request, cache misses per request, and context switches per request are each a single number, and each is a fingerprint of the workload’s character. If the candidate run’s fingerprint is within a few percent of the target’s on all four, the run is a defensible proxy. If one differs by a factor, the profile will differ in whatever code that counter governs.

A second check is stability. Running the same workload twice and comparing the folded stacks, as described in differential profiles, gives a lower bound on the noise, and any difference between an optimised and an unoptimised build must exceed that noise before it means anything.

The final check is adversarial: take the top function in the profile, construct a deliberate argument for why that function is hot in this run but not in the target, and test it. If the argument holds, the workload is wrong. The exercise is uncomfortable, which is why it is worth doing before the optimisation rather than after, when the work has already been justified by the profile it was based on.

A workload that cannot be made representative is not useless; it is a constrained result. Reporting a profile from a synthetic run as “the program spends 40% in the parser” states a fact about the run. Reporting it as a fact about the program requires the representativeness argument to be made explicitly, and the argument is usually shorter than the paragraph needed to hedge a claim that has none.