Differential Testing Between a Legacy System and Its Replacement

Replacing a legacy system requires answering whether the replacement behaves the same way, and the usual sources of truth are unavailable. There is no specification precise enough, the tests do not exist, and the people who knew the intent have moved on. Differential testing works around the missing oracle by treating the old system as the oracle and running both implementations on identical inputs. When they agree, the input is covered. When they disagree, the disagreement is the object of study.

The technique is old and general, used by compilers to compare optimisation levels and by protocol stacks to compare implementations, and its difficulty is entirely in the two edges: producing inputs that matter, and deciding what counts as agreement.

Building the input corpus

The corpus determines everything the test can find. Three sources complement each other.

Captured production inputs. Requests or batch records recorded at the system’s boundary, stored with their outputs, replayed against both implementations. This is the only source that reflects the real distribution: the odd encodings, the empty optional fields, the records that violate the documented format. It requires sanitisation, because the corpus contains personal and commercial data, and it requires capture at a stable boundary, which is the same instrumentation described in runtime dependency mapping.

Generated inputs. Property-based generators produce values at the edges of the input domain: empty strings, maximum lengths, zero and negative amounts, dates at the ends of the representable range, Unicode outside the basic plane. These find cases production never sees because callers never send them, until one does.

Hand-built boundary cases. The enumerated rules from the code: rounding at exactly half, the daylight-saving hour, the leap day, the record that exceeds a fixed-width field by one byte. These are cheap and they localise failures precisely.

A corpus is never complete, and the honest statement of coverage is the set of behaviours it reaches, not the number of cases. A thousand cases that all exercise the same branch prove no more than one case does.

When a divergence appears, the corpus should shrink around it. Delta debugging removes input fields, characters, or records while the difference persists, and the result is usually a five-line reproduction instead of a forty-field record. Minimisation is worth automating because the alternative is a triage queue full of large records that nobody reads. A useful by-product is that the minimised case often reveals which field the two implementations treat differently, which is faster than reading either implementation.

Corpus size has a second-order cost that shapes the design: the comparison runs two systems per case, and running a legacy system is slow because it was never built to be driven at test speed. A corpus of a hundred thousand cases that takes days to execute will not be run before every change, so the practical arrangement is a small corpus in the fast suite and a large one on a schedule, with the small corpus selected to cover every branch the large one reaches.

Defining agreement before looking at the output

The comparison must be specified before running anything, because the alternative is to look at each difference and decide, case by case, that it is acceptable. That process always concludes that the new system is fine.

Agreement is usually defined on a canonical projection of the output rather than the raw bytes. The projection removes fields that are legitimately different between two systems and keeps the ones that carry meaning:

• identifiers generated independently by each system are replaced by stable placeholders or removed
• timestamps are replaced by their ordering, or removed when the caller does not observe them
• collection ordering is normalised by sorting on the business key, unless order is part of the contract
• floating-point values are compared with an absolute and relative tolerance, or converted to a fixed-point representation
• whitespace, line endings, and key order in serialised documents are normalised

Every rule in that list is a claim that the difference is unobservable. Some are obviously true: nobody depends on the order of keys in a JSON object. Some are false in ways that matter: a report sorted by an unstable key produces a different pagination when a row is added, and a caller may depend on the order. The projection should be reviewed by whoever owns the consumer, not by whoever writes the comparator.

def compare(old_out: dict, new_out: dict, allow: dict[str, str]) -> list[str]:
    a, b = project(old_out), project(new_out)
    diffs = []
    for key in sorted(set(a) | set(b)):
        if key in allow:
            continue  # documented as unobservable, with a reason in the mapping
        if a.get(key) != b.get(key):
            diffs.append(f"{key}: legacy={a.get(key)!r} new={b.get(key)!r}")
    return diffs

The allow mapping is data, versioned alongside the tests, with a reason per entry. It is where the migration records its judgements: a difference that is accepted is an assumption, and assumptions that live only in a reviewer’s memory are the ones that break after the cutover.

Running both sides without contaminating them

Each case must be executed against a fresh state or against a state that both systems derive identically. A legacy system that reads a sequence, caches an entity, or leaves a lock behind makes the second execution differ from the first, and the resulting divergence says nothing about correctness.

The practical arrangement is one disposable database per run, seeded from the same fixture, with both systems pointed at separate copies. If the fixture is large, a filesystem-level snapshot of a seeded instance restores in seconds and avoids re-running the seed. Copy-on-write filesystem snapshots make this affordable in a way that a logical restore is not.

Time and randomness must be pinned identically on both sides. A system that calls the clock directly will produce different dates if the two executions cross midnight, and a run that takes hours will eventually cross it.

Concurrency is the harder case. Both implementations must be driven with the same interleaving to produce the same output, which is usually impossible. The workable compromise is to compare invariants, such as the final balance and the set of resulting rows, rather than intermediate traces, and to run the concurrent scenario many times looking for invariant violations rather than for differences between single runs.

Classifying divergence

Differences fall into four classes, and the classification is the actual deliverable of the exercise.

Defect in the replacement. The old behaviour is correct and the new one is not. These are fixed before cutover and added to the corpus as regression cases.

Defect in the legacy system. The old behaviour is wrong and the replacement is right. These are the most dangerous, because fixing them silently changes behaviour downstream. Each one needs an explicit decision, a note to consumers, and usually a period in which the new system reproduces the old bug until the fix can be coordinated. This is bug-compatible pinning at the level of two systems rather than two test runs.

Unobservable difference. Metadata, ordering, formatting. These belong in the allow list with a reason.

Harness artifact. The difference comes from the test setup: a fixture that was not reset, a network timeout, a container clock. These must be eliminated, not classified, because they erode trust in the whole comparison.

Triage needs a queue with an owner and a rule for how long a case can stay unresolved. A corpus that produces hundreds of unclassified differences is no longer a test; it is a backlog nobody reads.

What the method cannot establish

Differential testing inherits the old system’s bugs. Every case where both systems are wrong agrees, and the comparison reports success. That is acceptable when the goal is behaviour preservation and unacceptable when the goal is correctness; the two goals need different oracles, and a migration usually needs both, in that order.

It also proves nothing about inputs outside the corpus. Coverage should be reported in terms of behaviours reached, and the composition of the corpus should be reviewed: a corpus dominated by captured traffic will under-represent the rare inputs that only generated cases reach.

Finally, agreement on outputs does not establish agreement on side effects. Two systems can return identical responses while one writes an audit row and the other does not, or while one emits an event with a different key. The comparison must include the observable side effects, such as rows written and messages published, or it will certify a replacement that behaves identically only from the caller’s immediate perspective. Those side effects are what the write-path cutover depends on, and what the final decommissioning check has to account for when the old system’s outputs stop existing.