Deterministic Time and Randomness in Test Suites

A test asserts that a token expires in fifteen minutes. It sets a start time, asks the code under test for the expiry, and compares. If the comparison is against the real clock, the assertion holds or fails depending on how long the code took to run, and the failure is unreproducible because the inputs differed. Two globals cause almost all such failures: the clock and the random generator. Both are read through ambient interfaces, both are trivially replaceable with an explicit dependency, and both have subtleties that make the obvious replacement wrong in specific ways.

Two clocks, and only one moves forward monotonically

Operating systems expose at least two notions of time. A wall clock reports a calendar instant, which is what gets serialised into a timestamp, and it can be adjusted arbitrarily: a time synchronisation daemon stepping it, an administrator setting it, or a virtual machine being restored from a snapshot all move it, in either direction. A monotonic clock reports elapsed time since an arbitrary origin, never runs backwards, and exists specifically so that measuring a duration does not depend on the wall clock. POSIX names both, and the distinction matters in tests because measurement of elapsed time and the assignment of a timestamp are different operations that a single now() call conflates.

Concretely, a retry loop that computes a deadline from the wall clock can spin forever if the clock steps backwards during the test, and a cache with a time-to-live computed from the wall clock can keep an entry indefinitely if the clock steps backwards past the insertion time. A monotonic clock is not a complete answer either: on Linux the monotonic clock excludes time the machine spent suspended, so a test that asserts a duration after a suspend sees a smaller elapsed time than the wall clock suggests. A duration assertion is therefore only meaningful against a clock whose suspend behaviour is known.

The test-facing consequence is that the injectable interface should have at least two methods even when the application needs one. An interface exposing a single now() forces every duration computation through a clock that can jump, and no amount of test discipline repairs that.

Freezing time is not the same as advancing it

The simplest controllable clock returns a constant. That makes timestamp assertions deterministic and makes anything with a deadline impossible to reach: a cache entry never expires, a retry backoff never fires, a token never becomes invalid, because no time passes. A frozen clock turns a whole class of tests into no-ops, and the resulting gap is invisible unless something else tests the expiry path.

The useful design is a clock the test moves. The test sets an initial instant, exercises the code, announces that a chosen amount of time has passed, and checks the new state. The code under test must not sleep on the real clock for this to work, so the sleeping operation belongs in the same injected interface as the reading operation; a test that sets a one-hour timer must be able to complete in microseconds. Implementations vary from a counter the test increments by hand, through a virtual scheduler that runs timers in order until the queue is empty, to a library-level hook. The hand-incremented counter is easy to reason about and forces the test to state the timeline explicitly, which is usually the point.

The failure mode of an injected clock is asymmetry. If part of the code reads the injected clock and another part calls the language’s built-in time function, the test controls only half the timeline and the disagreement shows up as a negative duration or an expiry that fires immediately. Finding those sites is the same exercise as finding any other undeclared input, and the same discipline that Isolation Boundaries: In-Process, Subprocess, and Sandboxed Tests applies to files and sockets applies here.

Where a global reference becomes unfixable

Replacing the ambient clock at test time is a popular alternative to injecting one, because it requires no change to production code. The technique patches the module that provides the time functions, so that subsequent lookups through that module return controlled values. Its limitation follows directly from how binding works in most languages: a module that imports the function by name copies the reference at import time, so patching the origin module afterwards does not affect it.

The same fragility applies to a partial patch. If a program reads both a wall clock and a monotonic clock, patching only the first leaves duration computations on the real clock, and the test can then observe a timeline that is internally inconsistent. A patch that replaces a whole module’s surface avoids the problem but is only as complete as the surface it enumerates, and it will miss any code path that reached the operating system through a different binding — a different standard library module, a native extension, or a subprocess. A subprocess is the hard case: a patched clock is per-process, so a test that spawns a child has two independent timelines unless the child is also configured, and the mechanism for that is environment variables or a file, not a function call.

The trade is legibility against scope. Injection makes the dependency part of the signature, so a reviewer can see that a component reads the clock, and the compiler enforces that construction supplies one. Patching is invisible to the signature and therefore invisible to the reader, and it silently covers only what it happens to reach.

type Clock interface {
	Now() time.Time
	Sleep(d time.Duration)
}

// Manual advances only when the test says so, so a one-hour timer
// completes without waiting an hour.
type Manual struct{ t time.Time }

func (m *Manual) Now() time.Time { return m.t }
func (m *Manual) Sleep(d time.Duration) {
	m.t = m.t.Add(d) // time moves by the requested amount, not by wall time
}

Note what the interface deliberately omits: there is no method returning elapsed time, because a computation expressed as a subtraction of two Now() values is a duration only if Now() is monotonic, and leaving that arithmetic to a helper is how the wall-clock and monotonic cases get conflated.

Seeding as a reproducibility contract

Randomness has the same shape of problem with a simpler remedy. A language’s global generator is seeded once per process, usually from the system entropy source, so the sequence a test observes depends on the process it ran in — which, under the distribution discussed in Distributing Tests Across Workers Without Losing Determinism, means a test can pass on one worker and fail on another with no code difference. Constructing a generator per test and seeding it explicitly removes the ambient dependency at the cost of threading the generator through the call path.

A seed chosen as a constant makes the test fully deterministic and therefore fully frozen: the same inputs are exercised on every run, forever. A seed derived from the run and the test’s identity preserves variation across runs while keeping a given run reproducible, provided the derivation does not depend on the process. The two properties to preserve are that a failure report contains enough information to re-run exactly that case, and that re-running the single test in isolation produces the same values as it did inside the suite. Seeding from process identity, the current time, or the worker index breaks the second property and is the reason a “flaky” random test cannot usually be reproduced by re-running it.

A third source of variation is the iteration order of hash-based collections. Interpreted languages commonly randomise string and byte hashing per process to make collision attacks harder, which means the order of a set of strings differs between two runs of the same program. A test that asserts on the whole rendering of a set is comparing something the language deliberately made unstable. The fix is to sort before asserting, or to assert membership rather than order; disabling the randomisation through an environment variable makes the test pass reliably and hides the fact that the order is not part of the interface. Many other ambient values have the same character: identifiers derived from process ids or the machine name, temporary file names, and object addresses that leak into default string representations. None of them are stable, and any assertion that captures them is asserting on the environment rather than on the program.

Calendar edges that no clock injection fixes

There is a category of time-dependent behaviour that is not about reading the clock but about the calendar, and injecting a clock only helps if the injected value is chosen to be interesting. Daylight-saving transitions are the canonical case: at the end of the transition, a local wall time occurs twice, and at the beginning of it a local time that never occurs at all. Code that stores a local timestamp and later converts it can pick the earlier or the later occurrence, and both choices are defensible — which is why the behaviour has to be decided rather than discovered. The zone rules themselves are data, maintained by IANA and shipped as a database that changes whenever a jurisdiction changes its rules, so a test that asserts on a specific offset for a future date is asserting on a version of that database.

Durations across such a boundary need the same care. A difference between two local timestamps in the same zone is not the elapsed duration when a transition falls between them, and an hour of a day can be twenty-three or twenty-five hours long. Serialising as an offset-bearing instant, as RFC 3339 describes, keeps the elapsed arithmetic unambiguous; keeping a local time with no offset makes it ambiguous by construction. A test that pins the offset used inside the code under test can then be deterministic even while the zone database moves underneath it, at the price of never exercising the boundary — which is why the useful test asserts on the boundary itself, with the zone fixed for the duration of the test rather than taken from the machine.