Profile-Guided Optimization and Hot-Cold Code Splitting

Every optimization that asks “is this worth doing” needs a frequency estimate. Without one, the compiler assumes each branch is taken with equal probability, each block executes the same number of times, and each function is as hot as every other. Those assumptions are wrong by orders of magnitude in almost every real program: a function called once at startup and a function called a billion times in an inner loop are treated identically, and a branch that fails on one input in a million is assumed to fail half the time.

Profile-guided optimization replaces the assumptions with measurements. The program is built once with instrumentation or sampled with a profiler, run on representative input, and the resulting counters drive a second compilation. The transformation that matters most is not a specific optimization but the restructuring of the program’s layout around the frequency data.

What gets recorded

Two mechanisms produce the data, and they trade accuracy against overhead.

Instrumentation inserts counters. Edge counters or block counters are incremented at each execution, and indirect call sites record which targets they reached. The overhead is large: an instrumented build is substantially slower than an uninstrumented one, but the counts are exact. Instrumented builds also change the program’s timing characteristics, which matters when the workload is itself timing-sensitive.

Sampling uses a periodic interrupt to record the instruction pointer, producing a statistical estimate of where time is spent. The overhead is small and the estimate is unbiased for hot code, but cold code may be sampled zero times, and a function executed once leaves no trace at all. Sampling also attributes cost to the sampled instruction rather than to the caller, which makes call-graph reconstruction harder.

Both mechanisms must handle the same hazard: the profile describes the training input, not all input. A branch that is 99% not-taken in training may be taken on every call in a different deployment. The optimization is therefore a bet, and the bet is hedged by keeping the untaken path correct rather than by assuming it is dead.

Branch probabilities and the numbers they feed

An edge counter pair gives a branch probability directly: taken count over total count. That probability then flows into the block frequency estimate by a simple propagation — a block’s frequency is the sum over incoming edges of predecessor frequency times edge probability — solved as a flow problem over the CFG with the entry frequency fixed at one.

The resulting frequencies are the currency of the rest of the optimizer. They decide:

Inlining budgets. A call site on an edge taken 99% of the time gets a much larger size allowance than one taken once, which is the mechanism Inlining Decisions and the Cost Model Behind Them describes as frequency scaling.

Block placement. Blocks are ordered to keep the hot path contiguous, so the fall-through after a conditional branch is the likely successor. Static prediction heuristics typically predict forward branches as not taken and backward branches as taken, so placing the likely successor in the fall-through position matches them; even where a dynamic predictor exists, contiguity of the hot path reduces instruction fetch and branch-target buffer pressure.

Register allocation spill choices. A value used in a cold block can be spilled cheaply; one used in the hot block should stay in a register. Allocators that see frequencies weight spill costs by execution count, which changes the assignment substantially compared with unweighted costs.

Partial inlining and outlining. A function whose early blocks are a sequence of argument checks and whose body is one hot path is a candidate for splitting: the checks move into a separate cold function, the hot path becomes the entry point.

Splitting a function along its frequency profile

Hot-cold splitting partitions a function’s blocks into two regions by frequency and emits the cold region as a separate function or a separate section. The hot region keeps the fall-through path; the cold region is reached by a call or a jump to a distant address.

The benefit is concrete and comes from several mechanisms at once.

Instruction cache locality. The hot region is smaller than the whole function, so it fits in the instruction cache with fewer conflict misses. Error handling, assertion failures, and logging code no longer occupy lines that the hot loop needs. On a 32 KiB instruction cache, removing a few hundred bytes of cold code from each of a dozen hot functions is a meaningful fraction of capacity.

Register pressure in the hot region. Cold blocks often use values that the hot path does not, and liveness across the whole function forces those values to stay live. Splitting the function removes those live ranges from the hot region, which can reduce spills without any change to the allocation algorithm.

Branch predictor and fetch behaviour. A conditional branch whose taken target is far away costs a fetch redirect; making the cold path a call or an out-of-line jump keeps the hot path’s branches falling through.

/* The source shape that outlining targets. */
int parse(const uint8_t *p, size_t n, out *o) {
    if (!p || !o) return -EINVAL;             /* cold: argument checks */
    if (n > MAX_RECORD) return -EMSGSIZE;     /* cold: bounds */
    for (size_t i = 0; i < n; i++)            /* hot: the loop */
        o->buf[i] = p[i];
    return (int)n;
}
/* After splitting, the cold returns move to a separate outlined function;
   the hot entry tests one combined condition and falls through to the loop. */

The combined-condition trick is the second half of the transformation and is easy to overlook. If several separate checks each have a cold failure path, the hot path executes a branch per check. Merging the conditions into one test — with the detailed error selection moved to the outlined function, which re-tests when it is called — reduces the hot path to one branch. The semantics are preserved because the outlined function recomputes which error applies, and the cost of that recomputation is paid only on the cold path.

The limits of layout optimization

Layout is a global property and the compiler optimizes it locally. Function ordering, section placement, and inlining interact: moving a function into a hot section changes the distance between call sites and their targets, which changes the cost of the calls, which changes whether inlining was worth it. Linkers perform the final ordering, and a compiler that emits per-function sections lets the linker place them by profile. That is the mechanism behind function-section layout: each function goes in its own section, the profile data is attached as metadata, and the linker sorts by hotness.

The measurement problem is that layout changes what the profiler would have measured. Rebuilding with a profile collected from a differently laid-out binary is not a fixed point, and iterating the profile does not necessarily converge. A second round therefore measures a binary whose layout no longer matches the profile it was given, and the changes it produces can be small or negative rather than cumulative.

Cold-code sections also have a cost when the cold path is taken: the call to the outlined function saves and restores registers, and the code it executes is a cache miss by construction. For code where the “cold” path is taken on a meaningful fraction of inputs, splitting makes that path slower. The threshold that decides what counts as cold is therefore a policy with a real failure mode, and the usual answer is to split only blocks whose frequency is a small fraction of the function’s entry frequency.

Profile formats and the portability problem

A profile is only meaningful with respect to the code it was collected on. Instrumentation counters are indexed by CFG position, and any change to the source renumbers them. Compilers therefore either record the profile against a hash of the function’s control flow — so a stale profile is detected and ignored — or use a representation keyed by branch identity rather than position. Mismatched profiles are a common source of silent degradation: the optimization runs with probabilities that no longer correspond to the code.

The second portability issue is the workload. A profile from a benchmark suite describes the benchmark. Production deployments often collect their own profiles, which is why profile-guided builds are common for server software with a known workload and rare for libraries distributed to unknown callers. Context-sensitive or per-input profiles partially address this, at the cost of more compilation variants to manage.

The third issue is determinism. Two builds with the same profile must produce the same binary, which requires the profile-processing step to be deterministic — sorted, not hash-ordered — or the resulting layout differs between builds. This is a build-reproducibility requirement rather than an optimization one, and it is easy to violate with an unordered map in the wrong place.

Where static analysis still wins

Profile data is an input, not a replacement for reasoning. A branch whose condition is provably constant is optimized regardless of what the profile says, because the profile would be misleading if the program’s input changed. A call whose target is provably unique is devirtualized without speculation. The interaction is that profiles supply the fallback when static proof is unavailable, and a compiler that trusts a profile over a proof will produce code that is wrong rather than merely slow.

The most valuable combination is speculation guarded by a check. A virtual call that is monomorphic in the profile can be compiled with a type test and a direct call, falling back to the general path when the test fails. The hot path is fast because of the profile; the correctness comes from the test. That pattern — profile-guided guess, statically verified at run time — is what makes indirect call optimization safe, and it is why the profile’s role is to order the guesses rather than to license the transformation.

The same ordering logic applies to the transformations that follow layout. Unrolling, as Loop Unrolling, Trip Counts, and the Cost of Code Growth notes, is wasted on a loop that runs once, and only a frequency estimate distinguishes that case from a loop that runs a million times. The profile does not make the transformation legal; it decides where the compiler should spend a fixed code-size budget.