Calling Conventions on an Operand Stack

A native calling convention divides labor. The caller places the first several arguments in named registers, the callee saves what it will clobber, and the return value comes back in a designated register. The convention is a contract written down in an ABI document, and both sides are compiled against it. An interpreter has no registers to negotiate over and no callee prologue compiled ahead of time, so it needs a convention expressed entirely in terms of the operand stack and the frame layout. The design questions are the same ones an ABI answers: who removes the arguments, where the return value appears, and how a receiver is passed.

Arguments are pushed, and the callee’s locals begin there

The convention that stack machines converge on is simple to state. The caller evaluates each argument and pushes it, left to right, so the last argument ends on top. The call instruction carries a descriptor or an index that names the callee and fixes the argument count and types. On execution, the runtime creates the callee’s frame so that the n argument slots become the callee’s first n locals, in order.

The JVM encodes this in the method descriptor: (Ljava/lang/String;I)V means one reference and one int argument, returning nothing. The verifier checks that the operand stack at the call site has exactly those types in that order, so a call with a mismatched argument list never executes. Because the descriptor is verified rather than interpreted at run time, the interpreter’s call handler can assume the count and can compute the frame size from the callee’s own metadata.

The return instruction, ireturn or areturn, pops the single return value off the callee’s operand stack, tears down the frame, and pushes the value onto the caller’s operand stack in the position where the arguments began. So a call sequence is stack-neutral apart from the return value: whatever the caller pushed is consumed, and exactly one slot appears. That property is what allows expression compilation to treat a call as an ordinary node with a known arity.

The receiver in slot zero

Instance methods add one wrinkle. Before the arguments, the caller pushes the receiver. The call instruction then resolves the method against the receiver’s dynamic type, and the callee’s local slot zero holds the receiver. This is why the descriptor for an instance method does not mention the receiver: it is implied by the call instruction, not by the descriptor.

Slot zero being the receiver rather than a normal local has three consequences. First, this is addressable with the same one-byte local index as any other variable, so no special instruction is needed to read it. Second, a method that is not static always has at least one local, so max_locals is never zero for an instance method. Third, the constructor’s initialization rule can be enforced by verification: a constructor must call a superclass constructor before the object is used, and the verifier tracks this by treating slot zero as uninitialized until the invokespecial for the superclass constructor returns.

Formats without classes use a different placement. Lua passes arguments in registers starting at register zero of the callee’s window, with the function object itself available separately, and a method call in Lua’s syntax is sugar that inserts the receiver as the first argument. The difference is that Lua does not require the callee to know whether it was called as a method; a function can be called both ways and sees the difference only in how many arguments arrive.

Who pops the arguments

Two conventions exist in native code: caller-cleans, as in the cdecl convention, where the caller adjusts the stack pointer after the call, and callee-cleans, as in the stdcall convention, where the callee’s ret carries an immediate operand to pop the arguments. The trade is code size against flexibility. Callee-cleans saves the caller one instruction per call but makes variadic functions impossible, because the callee cannot know how many arguments were passed.

Interpreter frames make the question mostly moot. The callee’s frame is placed so that the argument slots are inside it, so when the callee returns, restoring the caller’s stack pointer to its pre-call value simultaneously reclaims the arguments and the callee’s frame. No explicit pop is needed on either side. The one place the distinction resurfaces is variadic calls: a call instruction with a dynamic argument count must pass the count somewhere the callee can read it, typically in the frame header or in a dedicated local slot, because the callee’s declared local count cannot express it.

Formats handle variadic calls in three ways. The JVM uses an array: the caller packs the variable arguments into a single array and calls a method whose descriptor takes that array. That keeps the descriptor static and verification simple, at the cost of an allocation per variadic call. WebAssembly requires the caller to place arguments in a fixed order and uses a distinct call_indirect for indirect calls, with no variadic form at all; languages needing varargs build them from a combination of fixed parameters and a pointer into linear memory. Lua passes an explicit count in the frame and lets a function declare that it accepts any number, so the callee reads the count from the frame header.

Indirect calls and the cost of dynamic dispatch

A call whose target is not known until run time goes through a different path. The operand stack holds the arguments and, for an instance call, the receiver. The runtime reads the receiver’s class, looks up the method by name and descriptor, and jumps to the resolved code. Caching the result is essential: the lookup is a hash or a linear search over a method table, and doing it per call would dominate dispatch cost.

Virtual calls in the JVM are resolved through a per-class virtual method table indexed by a slot number assigned when the class is loaded, so a resolved call becomes an array index rather than a name lookup. The slot number is stable for a class hierarchy because a subclass inherits its superclass’s slot assignments and appends new ones, which is the same layout guarantee that makes C++ virtual calls cheap. Interface calls cannot use a single table index because two unrelated classes implementing the same interface may assign different slots, so they need a per-call-site cache keyed by the receiver’s class.

A call site cache is usually one entry, holding the last class seen and the resolved target, with a comparison against the current receiver’s class on each call. If the comparison fails, the runtime falls back to the slow path and may replace the entry. Sites that see several receiver classes in alternation defeat a one-entry cache and benefit from a small polymorphic cache of two to four entries. This is the mechanism that in-place opcode quickening extends to the instruction stream, replacing a general dispatch with a specialized one that carries the resolved target.

Recursion, tail calls, and the missing frame

Deep recursion exposes the convention’s limits. Each call consumes one frame, and the stack’s maximum size is a fixed runtime limit, so a recursive method that descends deeply will exhaust it. The JVM specification places no requirement on tail calls: a return from a method that called itself still allocates a frame, so a tail-recursive method is not constant-space. The bytecode has no instruction that replaces the current frame with a new one for the same method.

Formats that do support tail calls encode them as a distinct call form. WebAssembly’s return_call and return_call_indirect reuse the current frame, so a tail-recursive function runs in constant stack space. The instruction’s semantics require the callee’s result type to match the caller’s, checked at validation time, because the frame being reused has the caller’s return type baked into its layout. Scheme implementations and the CLI’s tail. prefix serve the same purpose: the prefix instructs the runtime to discard the frame before transferring control.

The absence of tail calls changes what a compiler can do with a self-recursive function. Without them, a compiler can still transform tail recursion into a loop at the source level, which is what many language compilers do, but the transformation is a source-level rewrite rather than a bytecode mechanism. With them, the recursion can stay as written and the runtime handles the space.

What the convention must expose to the runtime

Any calling convention has to answer a question that native ABIs leave to debug information: where are the live references at this point? When a collection happens during a call, the collector must find every reference in every frame, including arguments, locals, and operand stack slots. The convention fixes this. Arguments occupy the low local slots, locals occupy the next slots, and the operand stack occupies the rest, so the live set is a prefix of each region. The frame header carries the counts, and the collector walks the chain from the current frame to the base.

That is why the convention is worth stating precisely rather than leaving implicit: the garbage collector, the unwinder, and the debugger all read the same layout, and each assumes the same ordering. A format that allowed arguments to be placed above the locals would need a separate map for each consumer. The JVM’s ordering is not an aesthetic choice; it is what lets a single frame descriptor serve all three.

How exceptions transfer control out of such a frame without the callee returning is the next problem, and it is handled by exception tables and unwinding.