Fixture Isolation and the Cost of Shared Test State

A test that passes alone and fails in the suite is not measuring the code. It is measuring the code plus everything the suite did before it. The distinction matters because the second measurement is unstable in a way the first is not: it depends on execution order, on parallel scheduling, and on whatever state the previous test failed to clean up.

Isolation is the property that each test’s result depends only on its own arrangement and the code under test. It is easy to state, hard to achieve, and worth a specific price — because every isolation strategy trades execution time for independence, and the wrong choice at either extreme produces a suite that is either slow or untrustworthy.

How state leaks between tests

Leaks have a small number of shapes, and each has a signature failure.

Process-global mutable state. A module-level cache, a singleton connection pool, a registry that tests register into. The first test populates it; the second sees the population. Failures appear as “passes in isolation, fails when run with the file above it”, and reordering the file makes them disappear.

External persistent state. Rows written to a shared database, files left in a temp directory, a queue with a message from the previous test. This is the leak that survives process restarts, so it accumulates: the suite is green on a fresh schema and red on a second run, and the difference is data nobody deleted.

Time. A test that captures now and compares against a computed expiry passes until the run crosses a minute boundary, a day boundary, or a leap second. The state is not stored anywhere; it is read from the environment at a moment the test does not control.

Randomness and hash ordering. A test that assumes a map iterates in insertion order, or that a UUID is unique across a fixed corpus, passes until the seed or the platform differs.

Identity and resource handles. A file descriptor not closed, a background goroutine still running after the test returns, a server bound to a port that the next test also wants. These produce failures in a later test, often in an unrelated file, which is the diagnostic nightmare case: the failing test is innocent and the guilty one already passed.

The common thread is that the test’s inputs include something outside its arrangement. Determinism requires controlling that something, either by resetting it or by making the test not depend on it.

Order dependence and why it hides

A suite with an ordering dependency is a suite whose tests form a chain. Test B needs A’s write to have happened. Run alone, B fails; run after A, B passes; run in a random order, B sometimes fails.

The dependency is invisible in B’s source. Nothing in B says “requires A”. The information lives in the accidental history of how the tests were written — usually A was written first and B reused its setup because it was convenient.

Test frameworks that randomize order exist precisely to surface these dependencies, and a suite that has never been randomized has an unknown number of them. The first randomized run typically produces a cluster of failures that are not defects in the code, and the work of fixing them is real work with no product value. It is still worth doing early, because the cost grows with suite size: a chain of dependencies discovered at two hundred tests is a day’s work, at five thousand it is a quarter.

Parallel execution surfaces the same dependencies plus a second class: tests that are individually isolated but contend for a shared resource. Two tests that each use the same fixed port, the same temp filename, or the same fixed database row pass serially and fail in parallel. Randomizing order does not find these; only running concurrently does. The catalogue of sources behind this class of failure, and the control that removes each, belongs to nondeterministic tests and flaky suites.

Four isolation strategies and their prices

The strategies differ in what they reset and what they pay.

Per-test construction. Every test builds its own object graph from scratch: a new in-memory store, a new service instance, a new clock. No reset logic exists, because nothing persists. The price is construction time per test, which is negligible for pure objects and significant for anything that opens a connection or reads a file.

Setup and teardown around each test. A fixture creates the resource before and destroys it after. The price is the cost of creation plus destruction, paid per test; the risk is asymmetric teardown, where the setup path has three steps and the teardown reverses two.

Transaction rollback. The test runs inside a transaction that is rolled back at the end. This is the cheapest way to isolate database state, since a rollback costs far less than a schema rebuild. It also has sharp limits: code that commits internally cannot be rolled back, sequences and other non-transactional objects are not restored, and a test cannot observe behaviour that only appears at commit time.

Snapshot and restore. A known-good state is captured once and restored between tests, by copying a data directory, restoring a database snapshot, or recreating a container. This is the only strategy that works for state the test process does not own, and it is the most expensive per reset — often hundreds of milliseconds to seconds.

The trade is legible: per-test construction costs microseconds and isolates everything the process holds; snapshot restore costs seconds and isolates everything on the machine. Choosing per-test construction for a unit suite and snapshot restore for a small number of end-to-end tests is the usual split. Applying snapshot restore to a suite of ten thousand unit tests produces a suite that takes hours, and applying per-test construction to an end-to-end test that needs a real database produces a test that cannot be written.

A third axis is scope of the fixture. A fixture shared across a file amortizes its cost over the file’s tests and introduces the risk that one test mutates what the next reads. A fixture scoped to a single test eliminates that risk at the cost of repeating the setup. Frameworks expose both scopes for this reason; the default should be the narrower one, and the wider scope should require a reason.

Making a shared fixture safe

Shared fixtures are sometimes unavoidable — a compiled binary, a migrated schema, a started server. The discipline that keeps them safe is to make the shared part immutable and the mutable part per-test.

import sqlite3
import pytest

SCHEMA = """
CREATE TABLE accounts (id TEXT PRIMARY KEY, email TEXT NOT NULL, status TEXT NOT NULL);
"""

@pytest.fixture(scope="session")
def schema_path(tmp_path_factory):
    # Session-scoped: create the schema once, never mutate it afterwards.
    path = tmp_path_factory.mktemp("db") / "test.db"
    with sqlite3.connect(path) as c:
        c.executescript(SCHEMA)
    return path

@pytest.fixture
def conn(schema_path):
    # Function-scoped: each test gets its own connection and rolls back on exit.
    c = sqlite3.connect(schema_path, isolation_level=None)
    c.execute("BEGIN")
    yield c
    c.execute("ROLLBACK")
    c.close()

The session fixture does the expensive, idempotent work once. The function fixture wraps each test in a transaction that never commits, so writes made by one test are invisible to the next. The combination gets most of the speed of a shared fixture with most of the isolation of a per-test one.

Two conditions make it correct. Nothing in the tested code may commit, since a commit escapes the rollback. And nothing may rely on state outside the transaction — a counter table updated by a trigger, a sequence, a cache in another process. Both conditions are worth a test of their own: a test that writes, rolls back, and asserts the row is absent catches the day someone adds a commit.

The cost of cleaning up versus not needing to

There is a design question underneath the fixture question. A component that opens a connection per call is harder to test than one that receives a connection; a function that reads the system clock is harder to test than one that receives a timestamp; a module that writes to a global log is harder to test than one that takes a logger. In each case, the testability problem is a dependency the component acquired implicitly.

Injecting the dependency moves the cost from the suite to the call site: one extra parameter, paid once, in exchange for tests that need no teardown at all. That is usually the better trade, and it is the reason testability arguments are legitimate design arguments rather than aesthetic ones. The counterargument — that a parameter added only for tests is clutter — holds when the dependency is genuinely global and stable, which is rarer than the argument’s popularity suggests.

Diagnosing a suite that fails only in aggregate

When a suite has become order-dependent, the repair sequence is mechanical.

Run the suite with randomized order and a fixed seed, so failures are reproducible. Bisect the failing test against the tests that precede it: run the failing test after each half of the suite until the polluting test is identified. The culprit is usually a test that writes without cleaning up, or one that leaves a background task running.

Then decide which of two fixes applies. If the polluter should clean up, add teardown and verify by running the pair in both orders. If the victim should not depend on the shared state, make the victim construct its own — the second fix is better, because it removes the dependency rather than managing it, and it does not require every future test to be careful.

The long-run fix is a policy: randomized order in continuous runs, parallel execution enabled, and a shared-state audit for any fixture wider than a single test. The isolation property then has a mechanical guard rather than a convention, which is the only kind that survives a growing team. How much of the suite should be fast enough to run on every change is a question about selection and budget, taken up in suite economics.