Reproduction Fidelity and the Sources of Nondeterminism

A fault is reproduced on a laptop, a fix is written, and the fault returns to production unchanged. The reproduction was real, the failure it produced was real, and the causal chain was different. Two different defects can produce the same observable symptom, and the second one survives because the first was eliminated. Recognising that a reproduction is a hypothesis about the mechanism rather than proof of it is the first step; the second is knowing which properties of the production environment can silently change which mechanism fires.

What nondeterminism actually consists of

A program is deterministic when its output is a function of its input. Real programs fail that condition in a small number of specific ways, and each has a characteristic signature in the shape of the failure.

Thread and goroutine scheduling is the largest source. On a single core the operating system’s scheduler is still free to interleave threads at any point where a preemption is permitted, so a data race between two unsynchronised accesses produces different results run to run. The number of interleavings grows exponentially with the number of concurrent operations, and the number of distinct schedules a testing run exercises is a vanishing fraction of them.

Time is the second source. Code that reads the wall clock, sets a timeout, measures an interval, or branches on a date takes input from outside the program. A ten-second network timeout that fires in a deployed system because a garbage collection pause delayed a response will not fire on a fast machine, so the error path that the incident exercised is never entered during local reproduction.

Random number generation is the third. Any use of an unseeded generator, or a seeded one whose seed derives from time or process identity, produces different inputs each run.

Hash-dependent iteration order is the fourth and the least obvious. In Go, iteration order over a map is deliberately unspecified and varies between runs; the specification states that the order is not specified and that it can change from one iteration to the next, which is a correctness guarantee about programs and a hazard for reproducing a bug that depends on which key is visited first. Rust’s default hasher is seeded from a source of randomness per instance for the same reason. A path that is only taken when a particular key happens to come first will appear intermittently, and the intermittency is a property of the runtime, not of the code that failed.

Input order at the boundary is the fifth. Requests arriving in a different order, with different concurrency, and with different sizes produce different states in shared structures such as connection pools, caches, and rate limiters. A bug that requires a specific warm state reproduces only when the same prefix of traffic is replayed.

The reproduction is a sample of a set

The reframing that resolves most of the confusion is that the failure is not one event but a set of executions, and each run samples one of them. A reproduction attempt produces a member of that set. Whether it produced the member that was observed in the deployed system is unknown, because that observation usually carries only the symptom and a partial trace.

Three outcomes are possible and they have different implications. The same mechanism fires and the same symptom appears, which is the good case, though it does not by itself prove that the local conditions are sufficient to explain the observed behaviour. A different mechanism fires and produces the same symptom, which is the dangerous case, because the investigation will converge on a fix for the wrong defect. No symptom appears, which is a negative result whose strength depends on how many samples of the execution set were exercised.

Increasing the number of samples is usually cheaper than increasing fidelity. Running the test suite under a stress scheduler, running the reproduction in a loop, or running it with an artificial thread-count increase all sample more of the interleaving space. This buys detection probability rather than correctness: a race that fires once in ten thousand schedules may still be missed, and the absence of a failure after a hundred runs is not evidence of absence.

Deterministic record and replay

The alternative to sampling is to record one specific execution and replay it exactly. The technique records the nondeterministic inputs at the boundary of the program — system call results, signal delivery, scheduling decisions, and the values returned by clock and randomness sources — and then re-executes the program with those values supplied from the recording instead of from the environment. Because the replay supplies the same decisions in the same order, a failure observed during replay is reproducible under a debugger.

The engineering constraint is that the recording must capture enough to constrain the execution and little enough to remain fast. Capturing every memory access is prohibitively expensive in time and space; capturing the outcomes of the operations that could have returned something different is the practical compromise, and it is what makes the technique usable on ordinary workloads. The recorded trace grows with the number of such operations rather than with the work done, so a long-running process with infrequent synchronisation records far less than its instruction count suggests.

Replay also has a limit that is easy to overlook: it reproduces the sequence of inputs, not the state of the world outside the program. A service that replays a recorded session against a database that has since changed will diverge. This is why replay tooling is most reliable for single-process programs, and why distributed replay requires recording the external responses of every peer, which is the same problem that reconstructing an incident from partial evidence runs into at a larger scale.

Making the program friendly to replay is a design choice that pays for itself. Passing a clock interface rather than calling the system clock directly, taking a random source as a parameter rather than reaching for the global one, and iterating deterministic collections when order affects behaviour all narrow the recorded boundary. The following shape is common because it costs almost nothing:

type Clock interface{ Now() time.Time }

type Expirer struct {
	clock Clock
	ttl   time.Duration
	seen  map[string]time.Time
}

// Production passes a clock reading the system time; a replay driver passes
// one returning recorded readings in order, so the timeout branch fires
// exactly when it did before.
func NewExpirer(c Clock, ttl time.Duration) *Expirer {
	return &Expirer{clock: c, ttl: ttl, seen: make(map[string]time.Time)}
}

func (e *Expirer) Expired(key string) bool {
	t, ok := e.seen[key]
	return ok && e.clock.Now().Sub(t) > e.ttl
}

The cost is indirection through an interface, one extra dereference per call, and the risk that a developer bypasses the abstraction and calls the global clock anyway. The benefit is that the failure’s dependence on time becomes explicit and testable rather than latent.

Widening the criterion instead of narrowing the input

A different strategy abandons the attempt to reproduce the exact event and instead asserts a weaker property that the incident violated. If the incident was a lost update, the property is that concurrent updates do not lose data; a test that hammers the same key from many goroutines asserts that property without needing the original interleaving. If the incident was a memory growth that ended in an out-of-memory kill, the property is bounded retention under sustained load, and the test can run against synthetic input at a smaller scale and for a shorter duration.

The trade is between specificity and sensitivity. A property test that checks an invariant can catch the bug and also catch unrelated bugs with the same shape; that is usually an advantage during debugging and sometimes a problem, because a test that fails for many reasons gives less information about which one occurred. Widening also loses the ability to assert on intermediate values, since the original ordering is gone.

A useful middle ground is to keep the original input and conditions and vary one environmental parameter at a time: number of worker threads, size of the heap limit, latency injected into a dependency. Each variant tests a hypothesis of the form “the failure requires this condition”. That is a discriminating test in the ordinary sense, and its outcome is more informative than a change of code.

Where reproduction stops being the right tool

Some faults are properties of the deployed topology rather than of the program. A latency problem caused by a load balancer’s connection reuse interacting with a backend’s keep-alive timeout exists in the arrangement, not in either component, and a single-process reproduction cannot exhibit it. A fault caused by a host whose CPU frequency governor changed exists below the process. For these, the reproduction has to be a scaled-down topology, and the fidelity question becomes which parts of the topology are load-bearing.

The signal that reproduction is the wrong tool is a sequence of successful reproductions that produce the symptom without explaining the magnitude or the timing of the incident. The symptom is easy to reproduce; the severity is not. At that point the useful question has changed from “can this be made to fail” to “what made this fail more often or more completely in the deployed system”, and the answer usually lies in concurrency, scale, or the state of a shared resource, all of which are measurable without reproducing the exact event.