Isolation Boundaries: In-Process, Subprocess, and Sandboxed Tests

Two tests that run in the same process share more than the objects a fixture hands them. They share the module cache, the locale, the current working directory, the open file descriptors, the signal dispositions, and every mutable global their libraries installed at import time. Frameworks provide fixtures to make the intended sharing explicit; nothing makes the unintended sharing go away. That gap is what isolation is for, and the levels of isolation available differ by orders of magnitude in cost, so the useful question is what each boundary actually prevents rather than which is best.

What leaks when tests share an address space

The cheapest runner design loads all test code into one process and calls each test in turn. Memory is shared, so the interpreter, the standard library, and the third-party dependencies are parsed and initialised once, which is the difference between a suite that starts in tens of milliseconds and one that starts in seconds per case.

The same sharing is what breaks. A cache populated by one test changes the timing or the result of the next. A monkey-patched function stays patched unless the patch is undone, and undoing it requires the patcher to remember the original binding. A library that reads an environment variable at import time reads it once, so a test that sets the variable before importing does nothing for a module already imported. A test that changes the working directory changes it for every later test, which silently redirects any relative path. A test that installs a signal handler changes the process’s disposition. A test that registers an atexit hook or leaves a background thread running affects the shutdown path of the whole suite.

None of these are exotic; they are the normal behaviour of a program that happens to be invoked many times in one lifetime. The mitigations available inside a single process are conventions plus a reset step: a fixture that snapshots and restores environment variables, a patch that is undone in teardown, an HTTP client that is constructed per test rather than stored globally. Every one of them is opt-in, and the failure appears as a test that passes alone and fails when the suite runs in a different order.

The process boundary

Spawning a fresh process per test removes all of the above at once, because the address space is new. Globals are reinitialised, file descriptors are inherited only if the parent explicitly passes them, the working directory is set at spawn, and a crash — a segmentation fault, an abort, a stack overflow — kills only the child. The parent observes an exit status rather than an exception, and an exit status carries more information: a nonzero code means the test reported failure, an abnormal termination visible as a signal means the test died, and neither is confusable with the other once the parent inspects the status correctly.

#include <sys/wait.h>
#include <unistd.h>

int run_case(const char *binary, char *const argv[]) {
    pid_t pid = fork();
    if (pid == 0) {          /* child: fresh address space */
        execv(binary, argv);
        _exit(127);          /* exec failed; distinguish from a test failure */
    }
    int status;
    if (waitpid(pid, &status, 0) < 0) return -1;
    if (WIFEXITED(status))   return WEXITSTATUS(status);
    if (WIFSIGNALED(status)) return 128 + WTERMSIG(status);  /* conventional */
    return -1;
}

The exit-code convention in that last line is the shell’s: a process killed by signal n is reported as 128 + n, which is why a runner can quote 139 for a segmentation fault without reading a core file. The cost is that the parent now needs to know the child is dead. A hung child holds the suite indefinitely, so the parent has to impose a deadline and escalate — send a termination signal, wait a grace period, then kill — which is a policy that has to be chosen rather than defaulted.

The price of the boundary is process creation. On a general-purpose kernel that means a new process descriptor, a new page table hierarchy, dynamic linker work if the binary is dynamically linked, and the initialisation of the runtime. It is small in absolute terms and enormous relative to calling a function, so a suite of ten thousand fast unit tests cannot pay it per test and a suite of a hundred integration tests comfortably can. Runners that support both modes let the coarse-grained level be the language-level grouping: Go, for instance, compiles each package’s tests into one binary and runs one process per package, so isolation happens at package granularity, and tests within a package run sequentially or concurrently in goroutines with no protection between them.

Cross-process isolation also imports a subtle problem from fork. In a POSIX process, only the thread that called fork survives in the child; if another thread held a lock at that moment, the child inherits the lock in its held state with no owner to release it. Runtimes with garbage collectors, thread pools, and internal locks therefore treat fork followed by more than exec as hazardous, which is why modern spawn implementations use a dedicated helper process or posix_spawn rather than the naive sequence.

Sandboxing and declared inputs

A process boundary isolates state. It does not isolate the filesystem, the network, or the rest of the machine. A test that reads a file from the current directory will keep passing on a developer’s machine long after the file was deleted from version control, because the file is there; a test that binds port 8080 collides with whatever else holds it.

Sandboxing addresses the first of these by running the test in a mount namespace or an equivalent restricted view, populated with exactly the inputs the build system says the test depends on. The design constraint is that the declaration has to be complete: a sandbox populated only with declared inputs turns an undeclared dependency into a failure, which is the point. It converts a latent portability bug into a visible one, and it does so at the moment the dependency is introduced rather than at the moment someone builds on a clean machine. Linux achieves the filesystem part with mount namespaces and the network part with network namespaces, and the process-level container most build systems use is a combination of several namespace types plus a separate filesystem root.

The cost is that any test with a genuine need to reach outside the declaration stops working, and the escape hatches differ per system: a test may be marked as requiring the network, or as running on the host, and those markers propagate into scheduling, because such a test cannot be run twice concurrently or sharded freely. Sandboxing also interacts with performance: a filesystem view built up per test costs more than one built per worker, and caching those views introduces its own invalidation rules.

Names that collide across processes

Three resources are shared even with a perfect process boundary: ports, temporary directories, and host identity.

A test that binds a fixed port fails when two instances run at once, which is exactly what sharding and CI parallelism produce. The portable solution is to bind port zero and ask the kernel which port was assigned, then pass the number to the code under test; a test that cannot do that because the port is in a configuration file is a test whose configuration must be generated per process. Temporary directories are easier because the platform provides a per-process one, but a test that uses a hardcoded path under the system temporary directory reintroduces the collision. Host identity matters for anything keyed on the machine name or the process id: identifiers generated from them differ between runs, which is fine for correctness and bad for anything that compares recorded output across runs.

These three are the practical reason isolation levels are chosen per test rather than uniformly. A pure function of its arguments gets nothing from a sandbox; a test that reads files, binds sockets, or mutates globals gets a great deal. Deciding which is which requires knowing what a test actually touches, and the machinery that answers that question from a build graph is the same one Change-Based Test Selection and Where It Stops Being Sound relies on. Where the boundary sits also determines what a failure means: in-process, a leaked global is a mystery that surfaces in an unrelated test, while across a process boundary the same leak is bounded, and Fixture Scoping and the Lifetime of Shared Test State covers the in-process half of that trade.