Sources of Nondeterminism That Make Tests Flaky

A test that fails one run in fifty and passes on retry is worse than a test that fails always, because a test that always fails gets fixed. The intermittent one gets retried, then quarantined, then ignored, and it teaches everyone who watches the pipeline that a red build may mean nothing. The damage is to the suite’s credibility, and credibility is the only thing that makes a failing test actionable.

Flakiness is not a mysterious property. Every instance has a source of nondeterminism, and the sources are a short list: the scheduler, the clock, the network, the filesystem, the random number generator, the order of iteration over an unordered container, and state that outlives the test. Each is detectable, and each has a control that makes the test deterministic rather than merely less likely to fail.

The retry reflex and what it costs

Retrying a failed test is the standard mitigation and it is a mistake with a specific cost structure. A test with a 2 percent per-run failure rate that is retried once has roughly a 1-in-2500 chance of reporting a failure, since both attempts must fail. The retry therefore converts a visible problem into an invisible one. The underlying defect, whether a race in the code or an ordering dependency in the suite, remains, and it now surfaces where there is no retry button.

Retries also destroy the signal needed to fix the problem. The failure rate is the diagnostic: a test failing 2 percent of runs under parallel execution and 0 percent serially implicates a shared resource; one failing only on a loaded machine implicates timing. A suite that retries silently never accumulates that evidence, and the failures that eventually escape are the ones that happen to be frequent enough to survive three attempts.

The defensible use of retries is narrow: a small number of tests that exercise a genuine external dependency whose failure is not the code’s fault, such as a third-party sandbox or a shared staging service, retried with the attempt count recorded and the retry rate alerted on. The retry rate is the metric; the pass rate is not.

Scheduling, parallelism, and shared resources

Two tests that each bind a fixed TCP port, write a fixed temp filename, or use the same database row pass in sequence and fail in parallel. The failure is not random from the scheduler’s perspective; it is a race for a name, and the winner is whichever test started first.

The controls are mechanical. Ports should be allocated by the operating system by binding to port 0 and reading back the assigned port, rather than chosen from a hardcoded range. Temp directories should come from the framework’s per-test directory facility, not from a fixed path. Database rows should be scoped by a per-test key, or the whole database should be per-worker.

Process-level sharing is the harder case. A test that starts a server for the whole file and another that starts a server for the whole file on the same port contend across files, and file-level parallelism turns that into an intermittent failure. The fix is either a port from the operating system or a fixture that owns the port for the process and is shared by every test that needs it.

A second scheduling source is internal to the test: a background goroutine or thread that the test starts and does not join. The test’s assertions run while the background work is still in flight, and whether the work finished first depends on machine load. Every test that starts concurrent work must wait for it to finish before asserting, and must have a mechanism for failing rather than hanging when it does not.

Clocks, timeouts, and the monotonic trap

Any test that reads the current time is a test whose behaviour depends on when it runs. The failure appears at a day boundary, a month boundary, a leap day, or when the wall clock is adjusted backwards by a time synchronization daemon.

The control is to inject the clock. A component that needs the time should receive a source of it, a function returning the current instant or an interface with a now method, rather than calling the system clock directly. The test then supplies a fixed instant or a clock it advances explicitly, and the assertion becomes exact.

Two details are easy to get wrong. First, elapsed-time measurement must use a monotonic clock, not the wall clock; the wall clock can move backwards, and a duration computed from two wall-clock readings can be negative. Language runtimes expose a monotonic source for exactly this reason, and a test that asserts on elapsed time should be asserting against it. Second, a fake clock must be shared by every component that reads time in the tested path, or the components disagree: one sees the advance, another does not, and the test fails for a reason unrelated to the code.

Timeout-based waiting is the other clock trap. A test that sleeps for a fixed interval and then asserts that an asynchronous operation completed passes on a fast machine and fails on a loaded CI runner. The replacement is to wait on the condition with a generous deadline, polling or subscribing, and to fail with a diagnostic when the deadline passes. The deadline is not a timing assertion; it is a bound on how long the harness will wait, and it should be long enough that reaching it always indicates a real problem.

class FakeClock:
    def __init__(self, start):
        self._t = start

    def now(self):
        return self._t

    def advance(self, seconds):
        self._t += seconds

class Expiry:
    def __init__(self, clock, ttl_seconds):
        self.clock = clock
        self.expires_at = clock.now() + ttl_seconds

    def is_expired(self):
        return self.clock.now() >= self.expires_at

def test_expiry_after_ttl():
    clock = FakeClock(start=1_000)
    e = Expiry(clock, ttl_seconds=30)
    assert not e.is_expired()
    clock.advance(29)
    assert not e.is_expired()      # boundary: still valid at ttl - 1
    clock.advance(1)
    assert e.is_expired()          # boundary: expired exactly at ttl

The test runs in microseconds and asserts the boundary exactly. The version that sleeps for thirty seconds asserts the same thing, takes thirty seconds, and fails when the machine is busy.

Iteration order and unordered containers

A test that asserts on the order of a map’s keys, a set’s elements, or a directory listing is asserting on an implementation detail that varies by language runtime, by hash seed, and by insertion history. Languages that randomize hash seeds do so deliberately, to prevent algorithmic complexity attacks, and the randomization makes the order differ between runs of the same binary.

The defect this hides is usually in the code: a serialization that emits a map in iteration order produces non-reproducible output, and a test that captured one ordering will fail intermittently for a change that is not a defect. The fix belongs in the code (sort before serializing), and the test then asserts on a stable order.

Where the order genuinely does not matter, the assertion should compare as a set or a multiset, not as a sequence. This is a case where the oracle’s shape decides the flakiness: asserting equality of sorted lists is both correct and stable, and asserting equality of the raw collections is neither.

Filesystem and network as ambient inputs

Two dependencies behave like the clock in that they are read implicitly, but differ in that their state is shared with every other process on the machine.

A test that writes to a fixed path inherits whatever the previous run left there, and the inheritance is invisible until the run order changes or a run is interrupted before cleanup. The control is a per-test directory created by the framework and removed afterwards, so the starting state is empty by construction rather than by cleanup discipline. Case sensitivity is a second filesystem trap: a test that passes on a case-insensitive filesystem can fail on a case-sensitive one, and the difference appears only on a different platform or a different mount. Paths that differ only in case should be a deliberate case in the suite, not an accident of the developer’s disk.

A test that reaches a real network endpoint depends on a service that may be slow, may rate-limit, and may change its response. The failure it produces is real information about the dependency and no information about the code, which makes it a poor gate. The controls are to replace the endpoint with a local listener the test owns, to record and replay responses so the bytes are fixed, and to keep a small number of tests that do reach the real service outside the gating tier.

The general rule for both is that a test should own every resource it touches: allocate it, use it, release it, so that the starting state of the next test is independent of what this one did. When ownership is impossible because the resource is genuinely shared, the test needs a way to detect the sharing rather than an assumption that it will not happen.

Randomness, seeds, and property runs

Any test that draws random values has a failure rate equal to the probability that a draw hits the failing region. A generator of arbitrary integers that occasionally produces a zero will produce a division-by-zero failure in a test that does not expect it, at a rate nobody predicted.

The control is a seed. A randomized test should draw from a generator seeded per run, with the seed recorded on failure so the run can be reproduced. Property-based frameworks do this by default, and the seed is part of the failure report rather than a hidden detail. A test that uses the language’s global random source without seeding is a test whose failures cannot be reproduced, which makes it undebuggable as well as flaky.

The subtle case is a library that seeds from the clock or from entropy at process start. A test that passes alone and fails in the suite may be seeing a different seed rather than different shared state, and the two have different fixes.

Detecting flakiness rather than tolerating it

The only reliable way to find a flaky test is to run the suite repeatedly under conditions that vary. A single green run says nothing about a 2 percent failure rate; a hundred runs of the affected file, with randomized order and parallel execution, surfaces most of them.

The practical protocol is to run the full suite N times with order randomization and record every test that failed in at least one run but not all. The list is the flaky set, and the failure rate per test is the prioritization. Tests that fail in every run are broken, not flaky, and belong in a different queue.

Once a flaky test is identified, the diagnosis is a bisection over the sources above. Does it fail serially? If not, the cause is parallelism or a shared resource. Does it fail with a single worker and randomized order? Then it is an ordering dependency in the suite, which is a fixture problem of the kind described in fixture isolation and shared state. Does it fail only at certain times of day? The clock. Does it fail only when the machine is loaded? A timeout or a scheduling assumption.

Quarantine as a bounded instrument

When a flaky test cannot be fixed immediately, quarantine is the honest alternative to retry: the test is moved out of the gating suite, tracked with its failure rate and its owner, and run in a non-blocking job. The gate stays trustworthy, the test is not silently deleted, and the defect remains visible.

The instrument fails when quarantine has no exit. A quarantined test that nobody owns is a deleted test with extra steps, and a suite with a permanent quarantine list has the same credibility problem as one with retries. The bound that makes it work is a count and an age: at most a small fraction of the suite, and nothing quarantined longer than a fixed period without a decision to fix or remove. The triage cost that makes this worth policing is set out in suite economics.

The longer-term answer is architectural. A test suite whose components are deterministic by construction (injected clocks, allocated ports, per-test state, seeded generators, joined threads) does not produce flaky tests to quarantine. That construction is the same discipline a simulator applies to a whole system, one component at a time, and it is the only mitigation that removes the source rather than the symptom.