Feedback Latency as Compounding Interest

A test suite that takes forty minutes to report a mistake is usually discussed as an inconvenience. It is better understood as a line item: every change to the system pays forty minutes, multiplied by the number of iterations the change needs, multiplied by everyone whose work waits behind it. That is interest in the strict sense — a recurring charge on work, caused by a property of the system rather than by the difficulty of the work itself.

The charge has a second component that is larger and less visible. Slow feedback changes how people work. When the loop is fast, a change is made small, tested, corrected, and merged, and a failure points at one edit. When the loop is slow, changes accumulate while waiting, because there is nothing to do but start the next thing. The failures that emerge then belong to a batch, and locating the cause inside the batch costs more than fixing it would have.

The arithmetic of a shared loop

The direct cost is easy to state and easy to underestimate. If a change requires three iterations and each iteration waits on a suite of duration $T$, the change costs $3T$ of wall-clock time in the loop. Some of that overlaps with other work, but the overlap is bounded: a developer cannot validate an edit before the result arrives, so the edit sits in an unfinished state, and unfinished work has a cost in context switching even when the person does something else in between.

The indirect cost is where the compounding happens, and it follows from a queueing identity rather than from psychology. Little’s law states that the average number of items in a system equals the arrival rate times the average time in the system: $L = \lambda W$. If changes arrive at rate $\lambda$ and each occupies the validation system for duration $T$, the average number of changes in flight is $\lambda T$. Double the suite duration and the average batch doubles, at constant arrival rate.

Batch size then feeds back into diagnosis cost. A failure in a batch of one has one candidate cause. A failure in a batch of ten has ten, and the tools for attributing blame inside a batch — bisection, per-commit test runs — each multiply the duration again by the number of commits examined, since bisection runs the suite $\log_2 n$ times at best. The result is that a slow suite does not merely delay feedback; it converts a linear cost into something closer to a quadratic one, because the delay produces batches and the batches produce repeated slow runs.

There is a hard limit as well. If the validation system has capacity $c$ — a fixed number of parallel runners — utilization is $\rho = \lambda T / c$. Queueing delay grows without bound as $\rho$ approaches one: for a single-server approximation the expected wait in the queue is $\rho T/(1-\rho)$, so at 90 percent utilization it is about nine times the service time and at 95 percent about nineteen times. The last few percent of capacity are the most expensive minutes in the system, and they are consumed by whichever tests happen to be slowest.

What slow feedback does to the code

The behavioural consequences are structural, not merely procedural, and they tend to be self-reinforcing.

Developers stop running the suite locally. The local loop is replaced by a push and a wait, which means pre-commit validation disappears and the suite becomes a gate rather than a tool. The gate then gets bypassed for “urgent” changes, or selectively disabled, and the tests that were supposed to prevent defects become a source of friction that people route around.

Test additions get expensive, so they stop. A new test that adds thirty seconds to a forty-minute suite is a hard sell, and the calculation is correct from the individual’s point of view: the marginal test costs the whole team its duration on every future run, while the benefit accrues to whoever breaks that behaviour. The suite stops growing in the areas that most need coverage — the recent, changing code — and grows instead where tests were easy to write.

Changes get batched, and batching is a design pressure. When review and validation are slow, there is an incentive to include “one more thing” in a change while the context is loaded. Large changes are harder to review carefully, so review quality falls, which raises the defect rate, which lengthens the loop’s importance. The coupling between latency and correctness is indirect but well worn.

Where the latency actually lives

Suite duration is usually dominated by a small number of test classes, and the distribution is heavy-tailed: a handful of tests take most of the time. The recurring causes are not mysterious.

End-to-end tests that drive a browser or a real network stack cost orders of magnitude more than unit tests, because each one pays process startup, page load, and network round trips. A suite of a few thousand such tests cannot finish quickly no matter how it is parallelized, and parallelization is bounded by runner count and by contention on whatever shared resource the tests touch.

Fixed waits are the second cause. A test that sleeps for a fixed interval to “let things settle” pays that interval on every run, including the runs where the condition was already true. The interval must be set for the slowest observed case, so the suite’s duration is set by the worst case rather than the typical one. The remedy is to wait on the condition with a deadline, which is both faster and more reliable.

import time

def wait_until(predicate, deadline_seconds=5.0, interval=0.01):
    """Poll a condition instead of sleeping a fixed interval."""
    deadline = time.monotonic() + deadline_seconds
    while time.monotonic() < deadline:
        if predicate():
            return True
        time.sleep(interval)
    return False

The third cause is setup that runs per test rather than per suite: creating a database schema, applying migrations, launching a container. If setup costs a second and there are two thousand tests, the suite spends over half an hour on setup regardless of what the tests assert.

Realism is the thing being traded

The reason suites become slow is that fast tests use doubles, and doubles do not exercise the real integration. A test with a mocked payment client cannot catch a change in the payment provider’s response shape; a test against an in-memory database cannot catch a lock conflict. The fidelity lost by each double is a real risk, and it is why the slow tests get written in the first place.

The trade is not binary. The useful decomposition is by failure mode: keep fast tests for the logic that can be verified in isolation, and spend the expensive integration budget on the specific boundaries where the double would lie — serialization formats, authentication, and transaction semantics. Contract tests occupy the middle: they verify that a consumer’s expectations match a provider’s actual responses, without requiring both to run in the same process. That keeps fidelity at the boundary while avoiding the full end-to-end cost.

The rule of thumb that follows is about count, not about category. End-to-end coverage should be proportional to the number of distinct integration paths, which is small, rather than to the number of behaviours, which is large. A suite with hundreds of end-to-end tests is almost always testing business logic through an interface that adds latency and removes diagnostic precision.

Flakiness is a separate charge

A slow suite and a flaky suite are different debts with a shared symptom, and conflating them leads to the wrong fix. Latency is paid in time. Flakiness is paid in trust: once a suite fails intermittently, its failures carry no information, and the rational response is to re-run until green. At that point the suite has stopped being a check and become a ritual, and the defects it was meant to catch pass through at whatever rate the re-runs allow.

Retries convert flakiness into latency, which is why they are popular and why they are a poor default. A retried test hides the race until the race becomes more frequent, at which point the retry count rises and the suite duration with it. The honest response is to quarantine the failing test, keep it out of the blocking signal, and treat its repair as work with a named owner, rather than to average over its inconsistency.

Paying this debt down

Latency is unusually tractable among architectural debts because the principal is visible: profile the suite, sort by duration, and a small number of tests will account for a large share of the total. The work is not intellectually hard, and the payback is immediate across every future change, which makes it a good early candidate in any repayment order — the reasoning developed in Sequencing Repayment Under a Fixed Capacity.

Two cautions belong with that optimism. First, the aggregate duration is the wrong target if the working loop is what matters: a suite that runs in ten minutes by sharding across forty machines has not reduced the charge if each developer still waits ten minutes and the failure attribution still spans a batch. The quantity to reduce is the time from edit to trustworthy verdict on that edit, which argues for running the affected subset first and the full suite later. Second, speed achieved by deleting tests is not repayment. The suite’s cost has to be reduced at constant detection power; if the removed tests were catching defects, the interest has been moved to a different account — the one described in Divergent Duplicates and the Cost of Two Truths, where defects surface downstream instead of at the edit.

The measurement discipline from Measuring Debt Without Inventing Numbers applies here with an unusual advantage: duration is one of the few debt-adjacent quantities that can be observed directly rather than proxied, because the build system already records it.