Suite Economics: Runtime, Feedback Delay, and Test Selection

A test suite is an asset with a running cost, and the cost is not the disk space or the maintenance hours. It is the delay between making a change and learning whether the change was wrong. That delay multiplies by the number of changes, and it sets a ceiling on how small a change can profitably be.

The economics are unusual because the benefit of a test is not proportional to its runtime, while its cost is. A test that catches one defect per year and takes ten minutes to run is a worse investment than ten tests that together catch the same defect in ten seconds — not because the defect matters less, but because the ten-minute test will be run less often, and a test that is not run detects nothing.

Feedback latency as the governing quantity

The relevant measure is wall-clock time from a change existing locally to a verdict arriving. It determines behaviour in a way that is easy to observe and hard to argue with: below roughly a second, a developer iterates without noticing the tool; at a few seconds, iteration continues but attention drifts; at a minute, the developer switches tasks and loses the context that made the failure interpretable; at ten minutes, the change is committed before the verdict arrives and the feedback loop is broken.

These thresholds are not precise, and they are stated as approximate because they vary with the work. The structural point is that a test’s detection value depends on the moment it reports. A test that would have caught a defect five seconds after it was introduced saves a fix; the same test reporting after the change was merged saves a revert, a review cycle, and a diagnostic session.

That gives the organizing principle for suite design: partition tests by the latency budget they must meet, not by the layer of the system they exercise. The usual partition is three tiers.

Inner tier, target under a second for the changed module, a few seconds for the file. Pure functions, parsing, formatting, state machines with injected dependencies. Thousands of cases, run on every save.

Middle tier, target under a minute or two. Tests that cross a process boundary with fakes, or that exercise a real database with a per-test schema. Hundreds of cases, run before every commit.

Outer tier, minutes to tens of minutes. End-to-end flows, real dependencies, concurrency and recovery scenarios. Tens of cases, run on every push and before merge.

The tiers exist in every suite whether or not anyone designed them; the design question is whether the assignment is deliberate. A suite where the inner tier accidentally contains database tests has an inner tier that takes forty seconds, and the cost is paid on every save by everyone.

What a test costs beyond its own runtime

Runtime is the visible cost. Three others dominate at scale and are usually ignored.

Setup and teardown. A test that takes 5 milliseconds but builds a schema that takes 400 milliseconds costs 405. Measuring the test body alone misattributes the cost; the fixture is part of the test. The strategies for reducing it — per-test construction, transaction rollback, snapshot restore — are priced in fixture isolation, and the choice between them is an economic decision as much as a correctness one.

Flakiness. A test with a 1 percent failure rate costs a triage investigation 1 percent of the time it runs, and the investigation typically costs far more than the test’s runtime — someone has to decide whether the failure is real, rerun it, and possibly halt a merge. At a thousand runs a day, a 1 percent flake is ten investigations daily. This is why flaky tests are an economic problem and not only a hygiene problem: their cost is dominated by the attention they consume, not the seconds they occupy.

Maintenance. A test that breaks on every legitimate refactoring costs a repair per refactoring. Suites coupled to implementation details — mock expectations on internal calls, assertions on incidental formatting — have a high repair rate, and the repair rate is invisible until the refactoring happens. The oracle and double choices that determine it are covered in stubs, fakes, and mocks.

Summed honestly, a suite’s cost per change is the sum over tests of runtime plus the failure rate times the triage cost plus the breakage rate times the repair cost, and only the first term is measured by the CI dashboard.

Parallelism and its ceiling

The obvious lever on the middle and outer tiers is to run tests concurrently. The speedup is bounded by the number of independent units of work, and that number is set by shared resources rather than by test count: a suite of a thousand tests that all serialize on one database has a parallelism of one.

Removing that serialization is the work. Per-worker databases, allocated ports, per-test temp directories, and no shared mutable global state raise the effective parallelism to the number of workers. The gains are close to linear until the machine’s CPU or I/O saturates, after which adding workers adds contention and flakiness rather than throughput.

Two second-order effects are worth knowing. Parallel execution surfaces latent isolation defects — tests that passed serially because they happened to run in a favourable order — so the first parallelization of a suite usually produces a batch of failures that are not defects in the code. And parallel execution makes timing-sensitive tests fail more often, because a loaded machine stretches every sleep and every timeout.

Test selection: running less, on purpose

Selective running executes a subset of the suite based on what changed. Three schemes exist, with different soundness properties.

Change-based selection. Map each test to the code it covers, from a coverage run, and run only the tests whose coverage intersects the changed lines. The scheme is sound for defects localized to changed code and unsound for anything else: a change to a shared data structure’s layout can break a test in a module that does not cover the changed line, and the selection will miss it. The usual mitigation is to widen the selection with reverse dependencies and to run the full suite on merge rather than on every intermediate commit.

Risk-based selection. Rank tests by historical defect yield and run the top slice within the budget. The ranking is learned from which tests have failed on which kinds of change. It is a heuristic with no soundness claim, and it is honest about that: it buys latency by accepting a detection probability below one.

Fail-fast ordering. Run the tests most likely to fail first, so a broken change is rejected in seconds even if the full suite takes minutes. Ordering does not reduce the total work; it reduces the time to the first verdict, which is the quantity that matters for a developer waiting.

All three trade completeness for latency, and the trade is only defensible if the complete run happens somewhere. A pipeline that runs only the selected subset on every commit and never runs the whole suite accumulates defects that no run would catch, and the accumulation is invisible because every individual run is green.

Determinism and its effect on the budget

A deterministic suite can be parallelized, cached, and partially re-run without ambiguity. A nondeterministic one cannot be safely parallelized, cannot be cached at the test level, and cannot support selection that trusts a previous result.

This is the economic argument for determinism, and it is stronger than the correctness argument usually given. Deterministic tests admit content-addressed caching: a test whose inputs, code, and environment hash to a previously seen combination can be skipped, because the result is known. That is a large multiplier on a suite with many unchanged tests, and it is unavailable to any suite with a clock dependency or an unseeded random draw.

The tiers are only real if their budgets are enforced. A table of the arrangement makes the contract explicit:

Tier	Trigger	Budget	Fail-fast	Scope
inner	on save	30 s	stop at first failure	changed module
middle	on pull request	5 min	stop at first failure	whole repository, parallel
outer	on merge	40 min	report all failures	end-to-end flows, real dependencies

The budgets are the design. A tier that exceeds its budget is a tier that will be skipped, and a tier that is skipped is a tier that does not exist. Treating the budget as a constraint to be enforced — failing the build when the middle tier exceeds five minutes, rather than watching it grow to fifteen — is what keeps the structure from decaying into a single slow stage.

Deciding what to delete

Suites grow monotonically because adding a test is cheap and removing one requires a judgement. The judgement is worth making, and the criteria are few.

Delete a test that cannot fail: one whose assertions are tautologies, one that asserts a value against itself, one whose body is entirely mocked such that no production code runs. These consume runtime and provide no signal.

Delete a test whose subject no longer exists: a test of a removed branch, a test that pins a behaviour the specification dropped. These fail for no defect and teach the team to update expectations without reading them.

Merge duplicates: five tests that each assert one field of the same call are one test asserting five fields, with lower runtime and the same detection.

Keep the test that is slow but irreplaceable — the end-to-end flow that is the only check that the system starts at all. Its cost is real and its value is real, and the honest accounting puts it in the outer tier where its runtime is paid once per merge rather than once per save.

The measure to optimize is not the test count or the coverage percentage but the time to a trustworthy verdict at each stage, and the fraction of defects each stage catches. Those two numbers, tracked together, describe a suite’s economics. Either one alone is a number that can be improved by making the suite worse.