Resident Programs and the Reentrancy Problem in DOS

A program that stays resident after exiting has no scheduler to run it. There is no process table, no time slice, and no context switch. Its only opportunity to execute is the moment another program’s execution is interrupted, which means it runs on someone else’s stack, with someone else’s registers saved, and in the middle of whatever that program was doing. That is a hard constraint, and the operating system’s own design makes it harder.

Terminating without releasing

The mechanism for staying resident is an interrupt that returns control to the shell while leaving memory allocated. The program calls the DOS terminate-and-stay-resident service with a value in a register indicating how many paragraphs of memory, 16-byte units, it needs to keep. DOS adjusts its own memory-allocation record so that the program’s block is marked as owned by a terminated process rather than freed, then returns to the parent.

Everything below the retained block stays allocated to the terminated process, including the program’s code, data, and stack. That memory is never reclaimed by DOS, because the allocation record says the owner has exited and the block is still in use. The practical effect is that a resident program permanently reduces the memory available to subsequent programs by its own size plus the size of the DOS structures that track it.

The paragraph granularity sets the floor on that cost. A resident program of any size occupies at least one paragraph, and the allocation record for it occupies memory in DOS’s own structures, so the smallest useful resident program still consumes more than its code and data. Programs competed for the few hundred kilobytes that remained after DOS and its buffers, and the resident programs that won were the small ones.

Running on a foreign stack

When the timer interrupt fires, the processor pushes the flags, the return address, and then the handler executes with whatever stack the interrupted program was using. That stack may be nearly full, may be in a segment the handler cannot assume anything about, and may belong to a program that will later examine it. A resident handler that pushes more than a few words risks overflowing it, and a handler that takes too long delays the interrupted program by exactly that duration.

The convention that developed was for a resident handler to switch to its own stack immediately. The handler saves the interrupted program’s stack pointer, loads its own, does its work, restores the pointer, and returns. That switch requires the handler’s stack to be in the resident block, which it is, and it means the handler’s stack usage is bounded by its own allocation rather than by the interrupted program’s.

The switch does not solve the deeper problem. The handler runs with interrupts enabled or disabled depending on what the interrupted code had set, and the flag state is part of what the handler must preserve. If the handler enables interrupts while the interrupted program had them disabled, a nested interrupt can occur at a point the interrupted program considered atomic. If the handler leaves them disabled for its whole duration, the timer’s own accounting drifts. The usual compromise is to disable interrupts for the shortest possible critical section and enable them for the rest.

Why DOS cannot be called from a handler

The operating system’s services are not reentrant. A program that calls the DOS function dispatcher sets an internal flag indicating that DOS is active, and a second call that arrives while the flag is set does not queue; it either fails or corrupts the state the first call is using. The reason is that DOS keeps a single set of internal variables: the current disk, the current directory, the transfer address for disk operations, the error state, and the pointer into the allocated-memory chain. Two nested calls would overwrite each other’s values.

The same applies to the firmware’s disk service, which uses the controller’s command registers and the DMA controller’s channel. A resident handler that decides to read a file when the timer fires will very likely fire in the middle of another program’s file read, and the two transfers will interleave their commands to the same controller. The result is not a clean error; it is data written to the wrong place, or a controller left in a state that no subsequent command can clear.

The documented escape is a flag that DOS maintains to indicate whether it is inside a service call. A resident handler checks the flag before calling DOS. If the flag is clear, the handler can call the service directly. If it is set, the handler must not call, and the alternatives are to set a pending-work flag and let the next timer tick retry, or to use the documented re-entry path that DOS provides for exactly this situation. The flag itself lives in the operating system’s data area below the one-megabyte boundary, and its address is version-specific, which is why the check is written against a published convention rather than against a fixed location.

That re-entry path works by allowing a second entry into the dispatcher only after the handler has saved and replaced a specific internal stack pointer, and it is documented in terms of the flag and the stack rather than as a general guarantee. Its existence is an acknowledgement that the resident-program model required a controlled exception to the non-reentrancy rule.

/* Structure of a deferred-work gate in a resident timer handler.
   dos_busy() reads the operating system's own in-call indicator; its
   address is version-specific, so it is abstracted here rather than named. */
static volatile int pending;

void timer_tick(void)
{
    if (dos_busy()) {          /* a service call is in progress */
        pending = 1;           /* record the work, do not re-enter */
        return;
    }
    if (pending) {
        pending = 0;
        do_deferred_work();    /* safe: the dispatcher is idle */
    }
}

The deferred-work pattern is the general answer. A handler that must call the operating system does the minimum work that cannot wait, records what it could not do, and returns. The recorded work runs on a later tick when the flag shows DOS is idle. The cost is latency: the deferred work happens at some point after the event that requested it, and the delay is bounded by how long the other program’s DOS call takes, which the handler cannot know.

The interrupt chain as shared state

A resident program that hooks an interrupt must preserve the previous handler and pass control to it. The saved value is a far pointer, and passing control is a jump rather than a call, so the previous handler’s return goes directly to the interrupted program. The handler that wants to do work after the previous handler has run must instead call the previous handler, then do its work, then return.

Which order is correct depends on what the handler needs. A keyboard handler that filters keys wants to see the key before the previous handler consumes it, so it runs first and either handles the key or passes it on. A timer handler that counts ticks wants the previous handler to have done its accounting first, so it calls the previous handler and then counts. Getting the order wrong produces a handler that never sees the input it was written for, or that double-counts time. The firmware usually has first claim on these vectors: the timer and keyboard services are installed before any program runs, so a resident hook is always layering onto a chain that already exists.

The chain also has a length limit that is not enforced. Each layer adds its own save, test, and restore, and the total latency of the chain is the sum. A machine with several resident programs hooked on the timer interrupt has a tick handler whose duration is the sum of all of them, and if that duration approaches the tick interval, the machine spends a substantial fraction of its time in handlers and the foreground program slows proportionally. The interval itself is fixed by the hardware’s timer, so the budget does not grow.

Removal is the final hazard. A resident program that wants to unhook must check that the vector still points to its own handler. If a later program hooked the same vector, the vector points to that program, and restoring the saved value would leave the later program’s handler in memory but unreachable, with its own saved pointer now dangling. The correct sequence is to walk the chain, find the entry that points to the handler being removed, and replace that entry with the saved pointer. That walk requires knowing the layout of every intervening handler, which is not standardised, so most resident programs documented that they must be removed in the reverse order of installation.