Compilation Budgets and Recompilation Policy

Compilation consumes the same processor the program needs. A runtime that compiles every method that looks hot will, on a program with many medium-hot methods, spend a substantial fraction of its cycles producing code and a smaller fraction running it. The mechanisms that keep that fraction bounded are budgets: limits on total compile time, limits per method, and limits on how many times a single method may be compiled.

Why a Budget Is Necessary

The naive policy of compiling whenever a counter overflows has no upper bound on work. A program that calls a hundred thousand distinct methods often enough to cross the threshold, or a program whose hotness is spread thinly across many methods, can trigger a compile for each. Compilation cost scales with method size and optimization level, so a policy that fires on count alone can be driven to spend more time compiling than executing by input that simply calls many things.

Two properties make the problem worse than a simple ratio. First, compilation happens on the critical path in the sense that it competes for cores; a background compiler thread is not free, it is a core the application cannot use. Second, compilation latency is not correlated with the benefit: a method that is hot for one second and then cold still costs a full compilation, and the benefit is realized only if the method is executed enough times after compilation to repay it.

A budget converts an unbounded policy into a bounded one. The runtime guarantees that compile time is at most some fraction of execution time, or at most some fixed amount per unit of wall-clock time, and once the budget is exhausted further compilation is deferred or refused. The thresholds the budget adjusts are the same counters that decide promotion between compilation tiers, which is why a budget cannot be designed independently of the tier ladder.

The Shape of a Compile Budget

Budgets appear in three forms, often combined.

A rate limit caps compile time per interval. The runtime tracks the time spent in the compiler and, when the rate exceeds a threshold, raises the hotness thresholds so that fewer methods qualify. The adjustment is feedback control: the thresholds are not fixed constants but variables driven by the observed compile ratio. Raising them under load is the standard response, and lowering them again when the compiler is idle prevents the policy from staying permanently conservative after one burst.

A per-method limit caps the optimization effort spent on any single method. Very large methods, including generated parsers, machine-written switch tables, and giant initialization functions, can take a long time to optimize for a benefit that may never materialize. Runtimes either refuse to compile methods above a size threshold at the optimizing tier, or compile them with a reduced optimization set. The threshold is expressed in bytecodes or in estimated machine instructions, and it exists because the cost of optimizing a method grows faster than linearly with its size once inlining and global register allocation are involved.

A total limit caps cumulative compile time for the process, or the number of compilations. This is the crudest form and the one most likely to be visible as a behaviour change: after the limit is reached, code that becomes hot later stays at whatever tier it reached. It exists mainly as a safety valve against pathological inputs rather than as a steady-state policy.

// Threshold adjustment driven by the observed compile-time ratio.
final class CompileBudget {
    private long compileNanos;      // time spent compiling
    private long runningNanos;      // time the program has been running
    private int threshold = 10_000; // initial back-edge threshold

    synchronized void recordCompile(long nanos) {
        compileNanos += nanos;
        double ratio = (double) compileNanos / Math.max(1, runningNanos);
        if (ratio > 0.10 && threshold < 1_000_000) {
            threshold *= 2;         // compile time above a tenth of run time: compile less
        } else if (ratio < 0.02 && threshold > 1_000) {
            threshold /= 2;         // compiler idle: allow earlier promotion
        }
    }
}

The constants in that sketch are illustrative rather than canonical; the structure is what matters. The budget is a control loop whose sensor is the compile ratio and whose actuator is the hotness threshold, and the loop’s stability depends on how quickly it reacts. Reacting on every compilation makes the threshold oscillate; reacting on a long window makes it slow to adapt to a workload change. Implementations typically sample the ratio at intervals of seconds and adjust in multiplicative steps.

When Code Is Compiled Again

Recompilation is triggered by evidence that the existing code is wrong for the current behaviour, not merely that it is old.

The first trigger is a profile that has changed. Code compiled when a call site was monomorphic may start receiving a second type, causing guard failures. The runtime counts failures per method, and past a threshold it recompiles with a profile that includes the new type. The recompiled code is less specialized at that site but has fewer failures, and the trade is usually favourable once failures exceed a small fraction of executions.

The second trigger is tier promotion. A method compiled at baseline may accumulate enough count to justify the optimizing tier. This is ordinary tiering rather than recompilation in the corrective sense, but it uses the same machinery and the same queue.

The third trigger is invalidation. A class is loaded, a method is redefined, or a dependency that compiled code assumed is broken. The existing code is invalid and must be replaced, and the replacement must be compiled from the current state of the world. Invalidation is the strongest trigger because it is not a performance judgement: the code is wrong, and the runtime must stop executing it. The machinery that finds and retires it is described under code cache invalidation.

The fourth trigger is a change in the compilation tier’s own configuration, such as a threshold flag or the selection of a different compiler. This is rare and usually means the runtime discards compiled code wholesale rather than recompiling selectively.

Bounding Recompilation

Each recompilation of the same method should cost more evidence than the last, or a method can oscillate. The standard mechanism is a per-method compilation count with an escalating threshold: the first compilation requires N events, the second 2N, the third 4N. The geometric growth bounds total work per method to a constant multiple of the first compilation, no matter how many times the method changes behaviour.

A cap on the count is the harder limit. After a small number of compilations, typically single digits, the runtime refuses further recompilation and leaves the method at whatever tier it reached. The method then runs with whatever specialization it has, accepting guard failures as a permanent cost. This is a deliberate choice in favour of bounded compiler work over optimal code, and it is the reason a method that alternates between two profiles eventually stops being recompiled.

Escalation interacts with deoptimization frequency in a way that can produce surprising behaviour. A method that deoptimizes on every invocation but is never recompiled runs slowly forever, and its slowness is invisible in profiles that only count time in compiled code. Runtimes that expose per-method compilation and deoptimization counts make the situation diagnosable; runtimes that do not leave it as an unexplained performance cliff.

Background Compilation and Its Deadlines

Compiling on a background thread removes compile latency from the requesting thread’s critical path, at the cost of two complications.

The first is that the profile can change while compilation proceeds. The compiler reads the profile at the start; by the time it finishes, the call site it specialized may have seen a new type. The compiled code is still correct because its guards check the assumption, but it may deoptimize immediately on installation. Runtimes handle this by re-reading the profile at installation and discarding a compilation whose assumptions no longer hold, or by accepting the occasional immediate deoptimization as cheaper than the check.

The second is that the queue can become the bottleneck. A burst of hotness, such as a batch job starting or a server taking its first traffic, can enqueue more methods than the compiler can process in the time the work takes. Priority then matters: compiling a caller before its callees, or a method with many callers before one with few, delivers benefit sooner. Some runtimes also allow the requesting thread to compile its own method when the queue is saturated, converting a queue delay into a synchronous cost charged to the code that caused it.

A budget and a queue are complementary. The queue determines the order in which work happens; the budget determines how much work happens at all. A runtime with a queue but no budget can be driven to spend all its time compiling, and a runtime with a budget but no priority ordering spends its allowance on whichever requests arrived first.

Measuring Whether the Budget Is Right

The observable that matters is the ratio of compile time to execution time, and it should be reported by any runtime that exposes diagnostics. A ratio near zero means the thresholds are too high and the program is leaving performance on the table; a ratio in the tens of percent means compilation is competing with the program.

The second observable is warmup time, the interval from process start until execution reaches steady state. Budgets interact with warmup because a budget that throttles compilation during startup delays exactly the compilations that would end the warmup. A policy that distinguishes startup from steady state, allowing more compilation early and throttling later, addresses both.

The third is the distribution of time across tiers, which requires knowing how much code is running interpreted, at baseline, and optimized. A program spending most of its time at baseline with an optimizing compiler available is either not hot enough to promote or blocked by a budget, and the two have very different fixes. Without the distribution, the budget’s effect is invisible, and the next policy change is a guess.