The Runtime as Reference Monitor: Mediation and the Trusted Base

A program that runs inside a language runtime is surrounded by code that decides what it may do. Every field read, every array index, every call to a library function passes through machinery the program did not write and cannot normally replace. That position, sitting between a program and everything the program can affect, is the same position a kernel occupies with respect to a process. The runtime is a candidate reference monitor, and whether it is a good one depends on properties that are easy to state and hard to satisfy.

Three conditions, stated before any implementation

Anderson’s formulation, written for operating systems in 1972, requires a reference monitor to be tamper-proof, to be invoked for every access, and to be small enough to be analyzed. The three conditions are independent. A mediator that checks every access but is itself reachable by the mediated program fails. A mediator that cannot be reached fails differently: it enforces nothing. A mediator that is correct but consists of a million lines of code cannot be analyzed, so its correctness is an assumption rather than a result.

Saltzer and Schroeder added the design principles that make the first condition achievable: least privilege, economy of mechanism, complete mediation, open design, separation of privilege, least common mechanism, and psychological acceptability. Two of these are architectural rather than advisory. Economy of mechanism keeps the analyzed core small. Least common mechanism says that data paths shared between two subjects must not be usable as a channel between them, which is an early statement of the covert-channel problem that covert channels still poses.

The runtime’s advantage over an operating system is that the mediated operations are richer and better typed. A kernel mediates open, read, and write on names and file descriptors; it has no idea that one of the bytes it copied was a method name. An interpreter knows the program’s structure, so it can mediate at the level of objects and calls, which allows policies that a kernel cannot express.

Where the boundary sits in a managed runtime

In a native process the mediator is mostly hardware: the MMU enforces page permissions, the CPU’s privilege levels separate user from kernel, and the kernel handles traps. The program’s own code executes directly, and nothing inspects its instructions.

A managed runtime inverts this. Source is parsed to an intermediate form, the form is verified, and execution proceeds by dispatching operations that the runtime implements. In a pure interpreter, the dispatch loop is the mediator: an array load cannot happen except by reaching the code that implements an array load, and that code performs the bounds check, the null check, and the type check before touching memory.

Verification is what makes the guarantee hold for compiled forms. Before a WebAssembly module may be instantiated, the engine validates the whole module: every instruction’s operand types must be consistent with the declared types, branch targets must be in range, and every memory access must be statically inside the declared linear memory or subject to a runtime bounds check. A module that fails validation is rejected before a single instruction runs. This is a whole-module property, checked once, which is why a validated module can then execute without per-instruction re-verification of its structure.

The same idea appears in bytecode verifiers for stack machines: the verifier computes, for each program point, the types that can be on the operand stack, and rejects the method if two control-flow paths disagree. Once that has been established, the interpreter’s checks reduce to a smaller set.

Operations that escape mediation

Completeness is the property that fails first in practice, because real runtimes contain paths that deliberately step outside the checked model. Each is a hole in the monitor, and each exists for a reason.

Just-in-time compiled code is the largest. Once a method has been compiled, the generated instructions access memory directly, with bounds checks retained only where the compiler could not prove them unnecessary. If the compiler’s reasoning is wrong, the check is absent where it was needed, and the program has escaped mediation. This is why JIT compilers are part of the trusted base rather than optimizations layered above it.

Unsafe or unmanaged code is the second. A block that may form raw pointers and dereference them can construct an object reference that points outside any allocation, and every later check on that reference passes because the reference itself is now the lie. Language-level sandboxing of untrusted code therefore cannot coexist with a general unsafe escape hatch in the same address space unless the escape hatch is itself verified.

Foreign function interfaces are the third. A call into C passes values across a boundary where the runtime’s invariants are not expressed, and the callee can hold onto a pointer after the call returns. The security question at that boundary is the one the WebAssembly import model treats in detail: which values cross, who owns them afterwards, and what the callee can do with them.

Signal handlers, longjmp, threads created outside the runtime, and memory-mapped files complete the list. Each is a documented path by which code executes or memory changes without passing through the dispatcher.

The price of a check on every access

A bounds check on every array read costs a compare and a conditional branch. On a modern out-of-order core the branch is predicted correctly almost always, so the marginal cost is small but not zero, and it is paid on the hottest operations in the program. This is the central trade: mediation completeness versus instruction count.

Runtimes respond by moving checks rather than removing them. Bounds checks can be hoisted out of loops whose trip count is provably within the array. Null checks can be merged with the load they guard, so that the fault of dereferencing null and the fault of a bad address are the same trap. Type checks at call sites can be cached: an inline cache records the receiver type seen last time and compares it, turning a lookup into a load and a compare.

A more aggressive technique removes the check by construction. If every object of a given shape is allocated inside a reserved region of known extent, then a reference is a tagged offset into that region, and the hardware can enforce the region’s bounds instead of the software enforcing per-object bounds. V8’s pointer-compression cage does this: heap objects live within a 4 GiB reservation, pointers are 32-bit offsets into it, and decompression is an addition. A corrupted 32-bit offset cannot address memory outside the cage, so the cage converts an arbitrary-write primitive into a constrained one. The same reasoning appears in fault isolation between tenants, where the reservation is what makes colocation defensible.

None of these techniques preserves completeness by itself. Each is a proof obligation: hoisting a bounds check requires proving the index range, and the proof is discharged by the compiler’s analysis, so the analysis becomes part of the trusted base.

What the trusted base actually contains

The trusted computing base of a managed runtime is every component whose failure breaks the isolation guarantee. Enumerating it is sobering:

• The parser and the verifier. A parser bug that mis-computes a length can produce a structure the verifier then approves.
• The compiler and its optimizer, including every analysis whose result removes a check.
• The runtime’s intrinsic implementations: array copy, string hashing, sorting, regular expression matching, and the numeric conversions.
• The garbage collector, which walks object graphs and writes forwarding pointers; a collector bug can resurrect a dead object or hand one tenant’s memory to another.
• The allocator, whose metadata is a target for corruption.
• The runtime’s own startup code, which reads configuration and constructs the initial object graph before any policy applies.

The size of this set is why the intrinsic functions and the parsers deserve as much scrutiny as the dispatch loop. The loop is small and heavily reviewed; a hand-written vectorized string search is neither.

The practical consequence for policy design is that a runtime sandbox should not be the only boundary. A defect anywhere in the trusted base converts the sandbox into decoration. The process boundary described in syscall filtering and process confinement remains underneath, and it is what limits the damage when the layer above fails.

Mediation at the granularity of the policy

A monitor that is complete but coarse enforces little. Two programs that both run in one process share the process’s file descriptors, its environment, and its address space; a policy that distinguishes them must be expressed inside the runtime, because the kernel sees only one subject.

This is the argument for capability-style interfaces: instead of deciding at each call whether the caller is allowed to write to a path, the runtime hands out a reference that carries the permission, and the callee’s authority is exactly the set of references it holds. The mechanism is the subject of capability references and ambient authority, and it converts the mediation problem from “check every operation against a policy” into “control which references exist”.

Both approaches need the same foundation: a boundary that the mediated program cannot cross, and a set of components small enough that their correctness can be argued. The remaining sections of this cluster are about the specific boundaries available, from the syscall filter to the type system, and about what each one can and cannot promise.