Speculative Inlining and Guard Placement

Inlining is the optimization that makes every other optimization better. A callee’s body in its caller’s context exposes constants, removes call overhead, and lets values stay in registers across what was a function boundary. A static compiler inlines what it can prove; an optimizing JIT inlines what its profile says usually happens, and pays for the bet with a check. The design question is how many checks the resulting code can afford.

Inlining Without Proof

A static compiler deciding whether to inline a virtual call must know the target. In a language with open classes and dynamic dispatch, the target is not known until run time, so the static compiler either leaves the call alone or devirtualizes it under a whole-program assumption that no subclass overrides the method, an assumption valid only if the program is closed and every class is visible.

A JIT is not closed, and it does not need to be. It can observe that this call site has only ever received objects of one class, inline that class’s method, and insert a check that the receiver is still of that class. If the check holds, the inlined body is the correct one. If it fails, the runtime transfers control to an interpreter or to less-specialized compiled code that performs the real dispatch. The optimization is valid not because the assumption is true but because the failure of the assumption is detected and handled.

This is the general shape of speculative optimization: assume the profile, check the assumption cheaply, and have a correct fallback. Inlining is its most valuable instance because the payoff is not local. An inlined callee can be constant-folded, its branches can be removed when their conditions become constants, its allocations can be eliminated by escape analysis, and its loops can be optimized with the caller’s knowledge of trip counts.

The Cost of a Guard

A guard is a comparison and a conditional branch to a slow path. On a predictable branch, that costs one cycle of issue bandwidth, or less if the compare fuses with the branch. A chain of guards each testing a different assumption costs proportionally, and the slow paths occupy code space and instruction-cache capacity even when they never execute.

The structure matters more than the individual cost. A guard on an object’s class pointer is a load and a compare, and the load is usually already needed for the inlined body’s field access, so the guard can be nearly free. A guard on a value’s range, needed when the compiler has assumed an integer stays within a bound to use a narrower representation, costs an explicit comparison. A guard on an array’s length, needed to eliminate a bounds check, costs a load and a compare, and eliminating a bounds check to add a length guard is only profitable if the bounds check would have been executed more than once in the same region.

Guards also constrain register allocation and scheduling. A guard’s slow path must be able to reconstruct the machine state at the guard point, so the values live at that point must be recoverable. The compiler therefore keeps a deoptimization map for each guard, which consumes metadata space and, more importantly, forbids some transformations: a value that has been optimized into a different form must still be describable in its original form at the guard.

What Gets Guarded

The assumptions a JIT makes fall into a few families, each with its own check.

Class assumptions. An inlined virtual call guards the receiver’s exact class or a class the compiler has proven equivalent. A field load that was converted to a fixed offset guards the object’s layout, which is a distinct check when layouts can change independently of classes, as happens when object shapes are assigned per allocation site rather than per class.

Array element type assumptions. Specialized array code for integers guards that the array’s element type is the expected one, and that the array is not a subtype with different storage. A store into a specialized array may also guard against a write barrier being needed, which depends on the array’s age.

Value range assumptions. Using a 32-bit operation where the language’s semantics require arbitrary precision requires a guard that the operand fits, and an overflow check that deoptimizes rather than wraps. The same applies to eliminating a null check when the compiler has proven non-nullness from a preceding check on the same value.

Branch frequency assumptions. Laying out a loop so that the exit test is at the bottom assumes the loop is entered at least once, which is checkable, or it assumes the test’s outcome is predictable, which is not checkable and must be handled by layout rather than by a guard.

// Guarded inlining of a virtual call, as an optimizing JIT would emit it.
// The profile said `shape` is a Rectangle at this site; the guard checks that.
#[repr(C)]
struct Shape { tag: Tag }            // class descriptor, first field of every shape

#[repr(C)]
struct Rectangle { head: Shape, width: f64, height: f64 }

fn area_of(shape: &Shape) -> f64 {
    if shape.tag == Tag::Rectangle {
        // The guard proved the layout, so the cast reads a Rectangle.
        let r = unsafe { &*(shape as *const Shape as *const Rectangle) };
        r.width * r.height          // inlined body, offsets known at compile time
    } else {
        deoptimize(shape);          // frame map for this guard; resumes interpreted
    }
}

The unsafe block is honest about the mechanism: the compiler is reading memory through a pointer whose type it has not proven, and the guard is the only thing making that sound. A compiler that omitted the guard, or that placed it after the load, would have a use-after-free-shaped bug rather than a performance bug.

The Guard Budget

An aggressive inliner can produce code with hundreds of guards, and at some point the checks cost more than the inlining saves. The trade is visible in three places.

Execution cost: every executed guard costs a branch, and guards inside loops execute once per iteration. Inlining a callee whose body is one field load into a hot loop, under a guard, trades a call for a branch, which is usually a win and sometimes not.

Code size: guards and their slow paths inflate the compiled method. A method that grows past the instruction cache’s capacity loses more to icache misses than it gains from inlining. Runtimes bound inlining depth and total inlined bytecode size for this reason, with limits expressed in bytecodes or in estimated machine instructions.

Deoptimization frequency: a guard that fails often is worse than no guard, because each failure discards the compiled frame and re-executes interpreted. A guard that fails on one call in a thousand is cheap; one that fails on one in ten means the profile was wrong and the specialization should not have been made. Runtimes count guard failures and use them to trigger recompilation with a wider profile, which is where recompilation policy meets inlining.

The interaction between these costs means the inliner cannot decide each call site independently. Inlining A into B makes B larger, which may push B past a size threshold and prevent B from being inlined into C. The inliner is therefore solving a global packing problem under a size constraint, and most implementations solve it greedily with heuristics: inline small methods, inline methods with a single call site, inline methods whose body becomes simpler after inlining, and stop at a depth or size limit.

When Inlining Is Unsound

The mechanism is sound only if every assumption is guarded and every guard’s slow path is correct. Three failures recur.

The first is an assumption that is not expressible as a guard. If the compiler inlines a method and then optimizes based on the observation that the method’s class cannot change, but the language permits class redefinition at run time, the observation can become false without any guard noticing. Runtimes that support redefinition handle it by invalidating compiled code that depends on a class, which is the mechanism described under code cache invalidation.

The second is a guard whose condition is weaker than the assumption. A compiler that inlines a method under a check that the receiver is a subtype of the profiled class, rather than exactly that class, has assumed that no subtype overrides the method. If one does, the inlined body is wrong. The correct guard is exact-class, or a subtype check combined with a separate check that no override exists.

The third is recursion. Inlining a recursive method requires either a proof that the recursion terminates, which is not available in general, or a depth limit that stops the expansion. Runtimes use the depth limit: recursive calls are inlined to a fixed depth and the remaining calls stay real calls, which preserves correctness at the cost of a partial optimization.

The fallback path is the load-bearing part of the whole design. Deoptimization must reconstruct the state the interpreter would have had, at the exact bytecode offset corresponding to the guard, including the values the inlined callee would have had in its own frame. That reconstruction is the subject of deoptimization frame state reconstruction, and it is where the cost of speculation is actually paid.