DOS Extenders and the Cost of Leaving Real Mode

A program that wants more than a megabyte of address space on a machine whose operating system runs in real mode has to build the environment it wants. The processor can enter protected mode, but the operating system cannot follow it there: DOS’s services assume real-mode addressing, its interrupt vectors are four-byte entries at physical zero, and its memory-allocation chain is described in real-mode segment terms. A DOS extender is the layer that keeps the operating system in the mode it understands while letting the application run in the mode it needs.

The transition itself is four instructions

Entering protected mode requires a table of descriptors, a load of the table’s base address into a register the processor consults on every address computation, and a setting of the protection enable bit. The descriptor table holds entries that name each segment’s base, limit, and access rights; the processor needs at least a code segment and a data segment to execute anything after the switch. On the 80286 the table register is loaded with a dedicated instruction and the enable bit is bit 0 of the machine status word, reachable through the instructions that read and write that word. The 80386 replaced the machine status word with control register 0, which the later instructions address directly.

The order matters because the instruction that sets the mode bit is followed immediately by an instruction fetch that is interpreted under the new rules. The code that performs the switch must therefore be at an address that is valid in both modes, and the segment registers must be loaded with valid selectors before or immediately after the switch. On the 80286 the sequence is: disable interrupts, load the descriptor table register, set the protection enable bit, then execute a far jump to a selector that names a code segment in the table. The jump flushes the prefetch queue and establishes the new code segment.

Coming back is the reverse, with an additional constraint. The processor must be in a state where the return is legal, which on the 80286 means clearing the mode bit and then executing a far jump to a real-mode address. The descriptor table is no longer consulted, so the jump target is a plain segment-offset pair.

; Enter protected mode on an 80286. The GDT must already be loaded and must
; contain a code descriptor and a data descriptor that cover this code.
        cli
        lgdt    [gdt_ptr]        ; base and limit of the descriptor table
        smsw    ax               ; machine status word
        or      ax, 1            ; set the protection enable bit
        lmsw    ax               ; the 80286 has no general mov to control regs
        jmp     far ptr pm_entry ; a far jump loads CS from the GDT
pm_entry:
        mov     ax, 0x10         ; selector for the data descriptor
        mov     ds, ax
        mov     es, ax
        mov     ss, ax
        sti

Two details in that sequence are load-bearing. The jump after the mode bit is set is not decorative: the processor may have prefetched instructions that were decoded under real-mode rules, and the jump discards them. The stack segment must be loaded with a valid selector before any push or call, because the stack descriptor’s limit is what makes the stack usable, and an interrupt that arrives with an invalid stack selector faults.

Where the interrupts have to live

In protected mode the interrupt table is not at physical zero. Its location is held in a register, and each entry is eight bytes rather than four, holding a selector, an offset, and attributes that name a gate type. A handler that runs in protected mode must be reachable through a selector that the table entry names, and the table itself must be in memory the processor can address.

The problem is that hardware interrupts still arrive, and the operating system’s handlers are real-mode code at real-mode addresses. The extender’s solution is to install its own protected-mode handlers for every vector, and have each one translate the call into a real-mode call. The translation is done by switching the processor back to real mode, executing the original handler, and switching forward again. That is expensive: a mode switch, a stack switch, and a return through the same path, for every interrupt.

The faster approach is to keep a small real-mode stub reachable from protected mode, so that the switch happens in a controlled place rather than in the middle of arbitrary code. The stub runs in real mode, calls the original handler, and returns to protected mode. The mechanism relies on a segment whose base and limit are arranged so that the same physical bytes are addressable both as real-mode segment-offset and as a protected-mode selector. That aliasing is the trick that makes the round trip possible without copying the handler.

Reflecting a call without losing the arguments

The application’s calls to operating-system services are the other half of the problem. A 32-bit program running under an extender calls DOS with a function number in a register and arguments in others, exactly as a real-mode program would, but the call arrives at the extender’s protected-mode handler. The handler must arrange for the real-mode operating system to execute the function with the same arguments, then return the results to the protected-mode caller.

The argument passing is complicated by the fact that real-mode DOS expects pointers as segment-offset pairs, and the application holds flat 32-bit addresses. A buffer above the first megabyte has no real-mode segment that can name it. The extender must therefore either copy the buffer into memory below the megabyte, or use a transfer buffer that the real-mode side can reach, and the copy costs time proportional to the buffer size. Programs that did many small disk reads through the extender paid for a copy on each one.

The return path has the same issue in reverse. A real-mode handler that returns a pointer, such as the address of a structure or a device parameter block, returns a segment-offset pair that names memory in the first megabyte. The extender must translate it to a flat address, and if the structure is later freed by the real-mode side, the flat address becomes invalid without the application knowing.

The documented interface for this translation is the DOS Protected Mode Interface, which defines a set of calls a protected-mode program uses to reach real-mode services and a set of data structures the two sides share. Its existence is an acknowledgement that the two modes could not be made transparent to each other, only bridged.

The address space the application actually gets

An extender does not give the application all of the processor’s memory. The first megabyte stays mapped for the operating system, the video memory, and the real-mode stubs, and the extended memory above it is available only through the extender’s allocator. The application’s address space is therefore a window whose size depends on how much physical memory is installed and how much the extender reserves for its own structures. The mechanics of that first-megabyte window, including the A20 gate that decides whether the region above it is reachable at all, are covered in segment arithmetic and the one-megabyte wall.

The allocator’s granularity matters. Protected-mode descriptors on the 80286 have a limit field of sixteen bits in units of bytes, so a single segment can name at most 65,536 bytes unless the granularity bit is set to make the unit 4,096 bytes, in which case the limit reaches just under four gigabytes in 4 KB units. The 80386 introduced a page mechanism and a flat model in which all segments have the same base and the full limit, which removes the segmentation arithmetic from the application entirely. An extender written for the 80286 had to manage segments explicitly; one written for the 80386 could present a flat address space and let the hardware’s paging do the translation.

That difference is why extenders of the later period could be simpler in their memory model while remaining more complex in their interrupt handling. A flat model means the application’s pointers are ordinary addresses, which removes the normalisation problem and the segment-boundary problem. It does not remove the need to return to real mode for every operating-system call, and it does not remove the need to handle interrupts that the operating system still expects to receive in real mode.

What the arrangement could not fix

The extender cannot make the operating system preemptive, because the operating system has no scheduler to hand control to. A 32-bit application under an extender is still a single real-mode process from DOS’s point of view, and it yields only when it calls a service that blocks. A program that computes for a long time holds the machine, and the extender has no mechanism to take it away. That is the same limitation that forces resident programs to defer their work rather than preempt whatever is running.

The extender also cannot protect the operating system from the application. In protected mode the application runs at a privilege level the extender chooses, and the operating system runs in real mode at a level the hardware does not distinguish. An application that writes to a real-mode segment through the extender’s translation can corrupt the operating system’s memory, and the hardware will not fault, because the write is a legitimate protected-mode write to a segment the extender mapped. The protection boundary is the extender’s descriptor table, not the processor’s privilege levels, and it is only as good as the table.

The final limit is that the environment is rebuilt on every program start. There is no resident protected-mode kernel that survives between programs, so each run loads the descriptor table, installs the handlers, and tears them down on exit. A later generation of extenders kept a resident portion, which is the point at which the extender starts to look like the operating system it was standing in for.