Test Discovery, Collection Order, and Import Side Effects

Test runners are usually described by how they execute: fixtures, parallelism, reporting. The phase that determines more behaviour than any of those is the one immediately before execution. Collection turns a directory tree plus a set of naming rules into an ordered list of test cases, and in every mainstream runner that step imports or compiles the test code. Import errors abort a whole file before a single test runs, module-level side effects happen before any fixture is entered, and the total count has to be settled before the first result can be reported.

Collection is a distinct phase that runs the language

A runner given a path does three things in sequence. It walks the path applying file-name rules, it loads each matching file in whatever way the language requires, and it inspects the loaded object for names that match the test convention. Only the third step is introspection; the second is ordinary program execution, with all the consequences that carries.

In Python, pytest’s defaults are that files named test_*.py or *_test.py are candidates, functions prefixed test_ inside them are tests, and classes prefixed Test with no __init__ method contribute their test_ methods. To find those functions the runner imports the module, which means the module body runs: constants are computed, registries are populated, environment variables may be read or written, and any expensive import at the top of the file is paid in full. A file that raises during import is reported as a collection error, not as a failing test, and no test in that file executes.

Go takes the compile-time version of the same idea. Files ending in _test.go are compiled into the package under test as a separate binary, and the runner then finds functions with the signature func TestXxx(t *testing.T). Nothing is imported at run time, so there is no module-level side effect during discovery, but package-level variable initialisation still runs before any test function, in the order the compiler determines from dependency order and, within a file, source order. JUnit’s platform splits the two concerns differently again: engines register with a launcher, and each engine decides what it can discover — a classpath scanner for the Jupiter engine, a directory scan for others. The platform’s contract is that discovery produces a tree of identifiers without executing test methods, which is a weaker promise than it sounds, because loading a class runs its static initialisers.

The practical result is the same across all three. Collection cost is proportional to the size of the test code plus whatever its imports drag in, and --collect-only style flags exist because that cost is worth paying separately from execution.

Naming rules are the selection interface

The naming convention is not cosmetic. Node identifiers built from file paths and function names are the handles that every other feature uses: selecting a subset, re-running failures, attributing a result to a case in a report, and caching outcomes across runs.

pytest node identifiers join components with ::, giving a string like tests/test_parse.py::TestHeader::test_rejects_duplicate. Go’s -run flag matches a regular expression against the test name, slash-separated for subtests created with t.Run, so -run 'TestParse/duplicate' selects one subtest. Both designs make renaming a test a breaking change for anything that refers to it by name, including the shell history of the person debugging it.

Parametrised tests complicate this. A test generated once per row of a table needs a distinct identifier per row, and the identifier is derived from the parameters. When parameters are integers or strings the derivation is obvious; when they are objects the runner has to invent something, and the invented name is usually positional. Duplicate names are resolved by appending a numeric suffix, which means the identifier of a case can change when an unrelated row is added earlier in the list. Anything that persists results by identifier — a cache of last failures, a dashboard, a quarantine list — inherits that instability.

Why the plan line has to come first

The Test Anything Protocol makes the constraint explicit. A TAP stream begins with a plan of the form 1..N, and a consumer reading the stream can treat a missing result as a failure only if it knows how many results to expect. A plan of 1..0 means the file was skipped entirely. The plan may be written at the end instead, but then a consumer cannot distinguish a truncated stream from a complete one until the stream closes, and a crashed runner produces a stream that looks complete up to the point of the crash.

Every runner that emits a summary of the form “N passed, M failed” has the same requirement. Counting during execution and printing at the end is possible, but any consumer that wants a denominator — a progress bar, a percentage, a harness that decides whether to continue — needs the total before the first test finishes. That is why collection is eager: the runner walks everything, loads everything, and builds the full list before dispatching the first case. A lazily collected runner starts faster and can report a failure sooner, at the cost of an unknown denominator and a harder time distributing work evenly.

The trade shows up in wall-clock behaviour on large suites. Eager collection pays a fixed cost proportional to the number of files, serially, before any parallelism begins. Lazy collection overlaps that cost with execution but cannot shard by count, only by file, and cannot tell a caller how many tests will run.

Order is stable but not an ordering guarantee

Collection order is deterministic in practice and unspecified in principle. Directory entries are typically sorted by name so that the same tree produces the same sequence on any filesystem, and files within a package are usually visited in name order. But the sequence is an artefact of the collector, not a promise in the language, and plugins routinely reorder the list — a randomiser plugin, a marker-based grouping plugin, or a hook that sorts by estimated duration.

That distinction matters because tests written against one order can silently depend on it. A test that assumes an earlier test populated a module-level cache passes under the collector’s order and fails when the suite is shuffled. Go exposes the check directly: go test -shuffle=on randomises execution order and prints the seed used, so a failure can be reproduced with -shuffle=<seed>. The flag exists because the dependency is otherwise invisible.

import ast, pathlib

def collect(path):
    """List test function names without importing the module."""
    found = []
    for file in sorted(pathlib.Path(path).rglob('test_*.py')):
        tree = ast.parse(file.read_text(encoding='utf-8'), filename=str(file))
        for node in tree.body:
            if isinstance(node, ast.FunctionDef) and node.name.startswith('test_'):
                found.append(f'{file}::{node.name}')
    return found

The collector above is deliberately weaker than a real one — it misses methods, parametrisation, and anything generated at run time — but it shows why a runner might parse instead of import. Parsing cannot trip over a module that connects to a database at import time, and it cannot be affected by the order in which imports resolve. It also cannot see tests that only exist after a decorator or a loop has run, which is the reason real runners import.

What breaks, and how it presents

The first failure mode is name collision. Two test files with the same basename in directories that are not packages produce the same import name, and Python’s module cache keeps the first one loaded; the second is reported as a mismatch or silently shadowed depending on the runner’s import mode. The fix is either unique basenames or __init__.py files that make the directories proper packages.

The second is import-time work that only succeeds in some environments. A test module that reads a configuration file at module scope fails collection on a machine without that file, and the report says the file could not be collected rather than naming the missing path as a test failure. Collection errors are also the reason a single syntax error hides every test in its file.

The third is generated cases. A runner that builds cases in a loop, or from a data file, has a count that depends on the data, so the plan line is only knowable after the generation code runs. When generation is itself parameterised by the environment, two runs of the same suite can legitimately report different totals, and any tooling that diffs counts between runs will report a change that is not a regression. The count is a property of the collection, not of the repository, and treating it as the latter is where most confusion about “missing” tests comes from.

Ordering and naming are also the substrate for everything downstream: a worker pool in Distributing Tests Across Workers Without Losing Determinism can only balance work it can enumerate, and the identifiers assigned here are what Change-Based Test Selection and Where It Stops Being Sound uses to map a code change back to a set of cases. A collection phase that produces unstable names quietly degrades both.