Ring Compression: Deprivileging a Guest Kernel Without Losing Its Own Protection

A guest kernel is written to run at the highest privilege level. It installs page tables, masks interrupts, programs the timer, and expects the processor to obey. The hypervisor needs that same level for itself. The conflict has one shape and several solutions, and the x86 protection model makes the solution choice consequential: the processor offers four privilege levels, and the guest kernel must occupy one that is strictly below the hypervisor’s.

The mechanics of the move are simple. The selector register that names the current code segment carries the current privilege level in its low two bits, so running the guest kernel at a lower level is a matter of loading a code segment with a lower descriptor privilege level. Everything difficult follows from what the guest kernel loses when it drops, and from the fact that the guest can read those two bits back.

Four levels, two usable gaps

The x86 protection model defines four rings, numbered 0 to 3, with 0 the most privileged. Most operating systems use exactly two of them: ring 0 for the kernel and ring 3 for user processes. Rings 1 and 2 exist in the architecture and are almost never used by a mainstream operating system, which is precisely what makes them attractive to a hypervisor.

Placing the hypervisor at ring 0 and the guest kernel at ring 1 leaves the guest’s user processes at ring 3. The guest then still has a genuine two-level hierarchy: its user code runs at ring 3, its kernel at ring 1, and the hardware enforces the boundary between them with the same page-table supervisor bit and the same gate descriptor privilege checks that a native kernel relies on. A page marked supervisor in the guest’s page tables remains inaccessible to the guest’s user processes, because the check is against the current privilege level being less than 3, and ring 1 satisfies that condition.

The alternative is to put the guest kernel at ring 3 alongside its own user processes. This is ring compression in the narrow sense: the guest’s two levels collapse into one, and the hardware can no longer distinguish guest kernel code from guest user code. Nothing about the guest’s address space changes, but every protection check the guest kernel expected the hardware to perform on its behalf now has to be performed by the hypervisor, in software, on every access that crosses the boundary.

The reason to compress at all is that a hypervisor running inside a host operating system cannot always claim ring 0 for itself and give ring 1 to the guest. When the hypervisor is a kernel module, the host kernel owns ring 0 and expects to keep it. Ring 1 is technically free, and some designs use it, but a guest kernel at ring 1 can still execute a set of instructions that a kernel module would rather it could not, and it can still be reached by the host’s own interrupt delivery in ways that complicate state separation. Compressing the guest to ring 3 makes the guest’s execution environment identical to that of an ordinary user process, which is the strongest guarantee a hosted design can offer and the most expensive one to enforce.

What ring 1 buys and what it still leaks

A guest kernel at ring 1 keeps its own protection hierarchy but loses direct access to everything the architecture reserves for ring 0. The instruction set divides cleanly along that line. Control-register moves, descriptor-table loads, HLT, INVLPG, and writes to the model-specific registers used for system call entry all require privilege level 0 and fault at ring 1. Those faults are the hypervisor’s interception points, and they are the ones trap-and-emulate is built to handle.

Two instruction families are subtler. Port I/O and interrupt-flag control are gated not by the current privilege level alone but by the I/O privilege level, a two-bit field in the flags register that only ring 0 code can modify. A guest kernel at ring 1 executing CLI or IN faults only if its I/O privilege level is numerically below its current privilege level. Set that field to 0 and every port access and every interrupt mask operation from the guest kernel faults, which is what a hypervisor wants; set it to 1 and the guest kernel gains unmediated access to every device on the machine. The field is therefore a switch the hypervisor must set to 0 and must never let the guest reach, and the guest’s attempts to write it are silently ignored rather than trapped, because POPF at a privilege level above 0 does not fault. It simply declines to load the privileged bits.

That silent decline is the leak that makes ring 1 uncomfortable. The guest kernel believes it has disabled interrupts. The hardware did not disable them. The hypervisor has to notice the intent some other way, either by intercepting the POPF instruction through code scanning or by never letting the guest’s interrupt state matter, which is why paravirtualized guests replace flag manipulation with an explicit call.

The privilege level itself is readable. A guest kernel at ring 1 can push its code segment selector onto the stack, or perform a far call, and read back a selector whose low two bits say 1. A native kernel at ring 0 would see 0. The guest can therefore detect that it is being virtualized with three instructions, which is a correctness problem for any design that promises transparency and a security problem for any guest that checks.

Gate descriptors and the syscall path

The most concrete consequence of deprivileging is what happens to the guest’s system call entry. On 32-bit x86, a user-to-kernel transition can go through a call gate, an interrupt gate, or a trap gate in the interrupt descriptor table. Each gate descriptor carries a descriptor privilege level, and the hardware permits an invocation only when the caller’s current privilege level is numerically less than or equal to that value. A gate with descriptor privilege level 0 is therefore unreachable from ring 3.

A native kernel installs its system call gate with descriptor privilege level 3, or uses an interrupt instruction whose gate has descriptor privilege level 3, precisely so user code can reach it. When the guest kernel moves to ring 1, the guest’s user processes at ring 3 still satisfy that check, so the gate works unchanged. The problem is the target: the gate must point at a handler that runs at ring 1, and the hardware performs a stack switch through the task state segment when the privilege level changes. The hypervisor must therefore own the task state segment that the guest’s own transitions use, or arrange for the guest to have one that the hypervisor can trust, because a task state segment written by a ring-1 kernel could redirect the stack pointer anywhere.

Ring compression removes the problem and replaces it with a worse one. If both guest kernel and guest user run at ring 3, the hardware sees no privilege change at the system call boundary, so no stack switch occurs, no gate check distinguishes the two, and the guest kernel cannot use the supervisor page bit to keep user processes out of kernel memory. Every guest memory access must then be checked by the hypervisor against the guest’s intended privilege, which means the hypervisor needs to know, for each guest instruction it executes, whether the guest thought it was in kernel or user context. That is state the hardware does not maintain when both levels are 3.

/* The check a compressed guest's memory access needs, in outline. */
static int guest_access_ok(struct vcpu *v, uint64_t gva, int write)
{
    uint64_t pte;
    int user_page;

    if (translate_guest(v, gva, &pte) != 0)
        return -1;                    /* let the guest take its own page fault */

    user_page = pte & PTE_USER;       /* U/S bit: 1 means user-accessible */

    /* Without a real CPL difference, the guest's mode lives in hypervisor state. */
    if (v->guest_cpl == 3 && !user_page)
        return -1;                    /* guest user touched supervisor memory */
    if (write && !(pte & PTE_WRITABLE))
        return -1;
    return 0;
}

The cost is proportional to how often the guest crosses its own boundary, and the boundary is crossed on every system call, every page fault, and every signal delivery. A native kernel pays a hardware privilege transition; a compressed guest pays a hypervisor check on each of those plus the software emulation of the transition itself.

System call instructions that bypass the gate entirely

Gates are the old path. The newer path is worse for a deprivileged guest. The SYSENTER instruction on Intel processors and SYSCALL on AMD processors do not consult a descriptor at all: they load the target address and segment from model-specific registers and jump there at privilege level 0. Those registers are writable only from ring 0. A guest kernel at ring 1 cannot program them, so a guest user process executing SYSENTER transfers control to whatever target the host programmed, at the host’s privilege level. That is not a virtualization inaccuracy; it is a direct branch into the hypervisor’s address space with the guest’s registers live.

The same hazard exists for SYSRET, which returns from ring 0 to a target whose privilege level comes from the STAR register. A guest kernel that cannot write those registers cannot use the fast system call path at all, which is why 32-bit paravirtualized guests used the interrupt-gate path and why 64-bit guests depend on hardware assistance rather than deprivileging. The hypercall interface is the other answer: the guest replaces the fast path with an explicit call into the hypervisor, which then performs the transition on the guest’s behalf at a known address.

There is a related leak in the segment registers. A guest kernel at ring 1 that loads a data segment with a descriptor privilege level of 0 is refused, because loading a data segment requires the selector’s privilege level to be numerically greater than or equal to the current privilege level. Guest kernels written for ring 0 routinely load ring-0 selectors, so a deprivileged guest faults on its own initialization code unless those selectors are rewritten. This is one of the reasons the first x86 hypervisors had to scan and patch guest code rather than rely on traps alone.

Why hardware assistance dissolves the problem

Every difficulty above comes from a single constraint: the guest must share the processor’s four privilege levels with the hypervisor. Hardware assistance removes the constraint by giving the guest its own complete set. Under the VMX execution model, the guest runs in a mode where rings 0 through 3 are all available to it, the guest kernel runs at ring 0, and the processor still intercepts the operations the hypervisor cares about.

The practical effect is that the guest’s own protection model is exactly what its author intended, gates and system call registers work as written, and the privilege level the guest reads back is the one it expects. Ring compression becomes a historical technique, retained in the designs that must run on processors without the feature and in the hosted products that cannot assume it.

What survives is the accounting. Deprivileging costs a trap per sensitive instruction; compression costs a software check per guest memory access plus emulated transitions; hardware assistance costs a configuration structure in memory and a defined exit path. The three are not interchangeable in cost, only in the problem they solve, and the rest of this cluster follows the boundary between them.