Deoptimization and Frame State Reconstruction

Optimizing compilers destroy information. A value that lived in a local variable slot becomes a register operand and then disappears into an addressing mode; a variable that was never modified becomes a constant; an object that was allocated becomes three fields in registers. Every one of those transformations is correct under the assumptions the compiler made, and every one of them makes it impossible to say, at an arbitrary point in the generated code, what the source-level state was.

Deoptimization is the mechanism that restores that state. When a speculative assumption fails, the runtime discards the compiled frame and resumes execution in the interpreter at the equivalent bytecode. The compiled code cannot simply return, because the interpreter needs the full frame, including locals, the operand stack, and every enclosing frame of inlined callees, as it would have existed at that bytecode. Producing that from registers and optimized-away values is the hard part.

What the Compiler Records

At every point where execution might leave compiled code, whether a failed guard, a call that can trigger class loading, or a safepoint where a garbage collection may occur, the compiler records a frame state. The record describes, for each value the interpreter would need, where that value currently is.

Three locations are possible. A value may be in a specific machine register. It may be in a specific stack slot in the compiled frame. Or it may be a constant the compiler folded, in which case the description holds the constant itself. A fourth case covers values that no longer exist as such: an object whose fields were scalarized into separate registers has a description that says “an object of class C with field f1 from register r1 and field f2 from stack slot s3”, and reconstruction must allocate a fresh object and fill it.

The description is not per-variable but per-frame-state, and frame states are shared where possible. Where two guards describe the same state the compiler emits one description and points both at it, so the number of distinct states in heavily optimized code is smaller than the number of guards. That deduplication is what keeps the metadata from swamping the code, and it also means a change to one point’s state costs a description rather than a whole new table.

/* A frame state as a compiler emits it: a chain of virtual frames, innermost first. */
struct frame_state {
    uint32_t bci;                  /* bytecode index to resume at */
    struct method *method;
    uint32_t nvalues;              /* locals plus operand stack entries */
    struct value_desc *values;     /* each: REG(r) | SLOT(s) | CONST(c) | OBJECT(...) */
    struct frame_state *caller;    /* the frame this one was inlined into, or NULL */
};

/* Reconstruct one virtual frame from its description. */
struct frame *materialize(struct frame_state *fs, struct compiled_frame *cf) {
    struct frame *f = push_interpreted_frame(fs->method, fs->bci);
    for (uint32_t i = 0; i < fs->nvalues; i++) {
        struct value_desc *d = &fs->values[i];
        switch (d->kind) {
        case REG:  set_slot(f, i, read_register(cf, d->index)); break;
        case SLOT: set_slot(f, i, read_stack(cf, d->index));    break;
        case CONST: set_slot(f, i, d->value);                   break;
        case OBJECT: set_slot(f, i, rebuild_object(cf, d));     break;
        }
    }
    return f;
}

The chain is what makes inlining reversible. An inlined callee has no frame of its own in the compiled code, so its state is described as a virtual frame whose caller is the frame it was inlined into. Deoptimizing at a guard inside the callee produces two interpreted frames, one for the callee and one for the caller, exactly as if the call had never been inlined. A deoptimization inside a doubly inlined body produces three. The runtime walks the chain and materializes each, and the resulting interpreted stack is indistinguishable from one produced by never having compiled the code.

Where Deoptimization Can Occur

Not every instruction can deoptimize. The points are restricted to places where the frame state is known and where the runtime can take control safely.

Guards are the obvious case: the slow path of a guard is a deoptimization point by construction, and the compiler emits the state description at that branch. Safepoints are the second case: a point where the runtime may need to stop the thread, typically at a backward branch or a call, where the frame state is fully described so that a garbage collector can find every live reference. The same description serves deoptimization, which is why compilers place them at the same points.

A third case is the entry to code that cannot be compiled speculatively at all. A method that is executing for the first time after a guard failure resumes interpreted, and the interpreter may re-enter compiled code later through a normal call.

The restriction matters because a deoptimization needs the machine state at the guard to be recoverable. If the compiler has reordered a store past the guard, or has moved a value into a callee-saved register that the slow path does not preserve, the state description is wrong and reconstruction produces a frame with garbage in it. Compilers enforce this by treating frame states as uses of the described values in the intermediate representation: a value that appears in a frame state cannot be reordered past the guard or optimized away without updating the description.

The Cost of Getting It Wrong

A deoptimization that reconstructs the wrong state is not a performance bug; it is a correctness bug that manifests only under specific timing. The classic shape is a guard that fires after a value has been modified but before the modification has been recorded in the description, producing an interpreted frame that resumes with a stale local.

A subtler failure involves objects that were scalarized. If escape analysis proved an object never escapes and replaced it with its fields in registers, reconstruction must allocate a new object. The allocation can trigger a garbage collection, which can trigger further deoptimizations or safepoint operations, which means deoptimization must be reentrant-safe. Runtimes handle this by performing the allocation through the normal allocation path with the thread in a state where a collection is permitted, rather than by hand-building objects outside the allocator.

A third failure is a guard placed where deoptimization is impossible, inside a region the runtime cannot interrupt such as a hand-written assembly stub or a section holding a lock. Runtimes avoid this by not placing guards in such regions, which sometimes forces the compiler to abandon an optimization rather than speculate inside it.

Frequency, and Why It Is Not Always a Bug

A deoptimization is expensive. It must unwind the compiled frame, allocate interpreted frames, fill them from registers and memory, and resume interpretation. The cost is orders of magnitude larger than the guard that triggered it, which is why a guard that fires in a hot loop is a serious problem and a guard that fires once per method invocation is not.

Frequency is the diagnostic. A method that deoptimizes thousands of times per second is cycling, and the runtime’s response is to recompile it with a less aggressive profile or to stop speculating at that site. A method that deoptimizes once when a new class is loaded is behaving correctly: the profile it was compiled against was accurate until the program changed shape, and the recompilation that follows will reflect the new shape.

Runtimes expose the count of deoptimizations, and the pattern is informative. Many deoptimizations at the same guard mean the profile was wrong. Many deoptimizations at different guards in the same method mean the method is genuinely polymorphic and should not be specialized. A single deoptimization at startup followed by none means the runtime learned the right profile on the second attempt.

Interaction with the Rest of the System

Deoptimization is the hinge between speculation and safety, so it touches nearly every other mechanism. The tier ladder uses it to fall back from optimized to baseline code. The inline cache supplies the profile that the guards test, and each guard failure is evidence that the cache’s record was incomplete. The code cache must keep a method’s metadata alive as long as any frame could deoptimize into it, which means metadata lifetime is tied to stack contents rather than to the method object, and the metadata outlives the code whose invalidated body still has frames pointing at it.

The reconstruction machinery also serves purposes other than speculation. A debugger inspecting an optimized frame needs the same information to display local variables, and a garbage collector needs it to find references in registers. That is why the description is designed as a general map from machine state to source-level state rather than as a deoptimization-specific structure: the same data serves three consumers, and a compiler that records it for one gets the others nearly for free.