Characterization Tests: Pinning Behaviour Nobody Designed

An ordinary test asserts a property the author believes the code should have. A characterization test asserts a property the code already has. The distinction is not cosmetic: when the two disagree, an ordinary test reports that the code is wrong, and a characterization test reports that the maintainer’s understanding of the code was wrong. On a system with no tests and no reliable specification, that second message is the only one available.

The purpose of the exercise is not coverage. It is to create a fixed point. Before any change, whether a seam, a refactor, an extraction, or a replacement, there must be a recorded answer to “what does this unit produce for this input”, so that a later change can be classified as intended or unintended. The test is a change detector, and it is deliberately blind to whether the behaviour it detects is correct.

Recording behaviour that already exists

The procedure is mechanical. Choose a unit small enough to execute in isolation, supply it with inputs, run it, and write down what came out. Repeat until the unit’s observable surface is covered. Then run the whole set against the unmodified code: every test must pass, because every expectation was copied from an actual run.

That last step is the safety property. A characterization test that fails on the current code is not a characterization test; it is a hypothesis, and either the harness is wrong or the extraction of the observation is lossy.

One practical consequence: the first run of a characterization test is expected to fail, and its output is the expectation. Writing the test means running the code with a placeholder assertion, capturing the actual value, and pasting it in. Doing this by hand for a hundred cases is slow and error-prone, so the capture step is usually automated: run the case, serialise the result, write it to the expected-value file, and only then assert. The assertion exists to catch the next change, not to prove anything about the first run.

There is also a scope question that the word “unit” conceals. Small units pin cheaply and fail precisely; large units pin expensive behaviour with a single assertion that says only “something changed”. For a legacy batch job, the unit may be the whole job run against a fixture database, and that is legitimate when nothing smaller can be executed. The trade is diagnostic quality: a whole-job golden master tells the maintainer that a refactor altered a rounding rule, but not which of the forty intermediate steps did it, so the next step is to subdivide the golden master into per-stage outputs until the failure localises.

In practice the difficulty is never the assertion. It is getting the unit to run at all. Classes with constructors that open sockets, functions that read the system clock, and static initialisers that query a database all resist instantiation. This is where seams earn their keep: an object seam lets the unit be constructed with inert collaborators, and a link seam lets the real module be replaced entirely for the test build. Characterization work and seam work alternate, because each test attempt reveals the next dependency that has to be broken.

The oracle problem and bug-compatible pinning

For a normal test the oracle is the specification. Here there is no specification, so the oracle is the implementation. This produces a well-known hazard: if the current behaviour is a bug, the test now protects the bug. A later, correct fix will be reported as a regression by the suite.

There is no automatic way to distinguish intended from accidental behaviour. What can be done is to record the decision. Mark the test, or the assertion, with the reason it exists and whether a human has judged the behaviour deliberate. The test then serves two purposes: it fires if the behaviour changes, and it carries the note that changing it was considered and accepted or considered and rejected. Without that annotation, the suite becomes a wall of assertions with no ranking, and the first person under deadline pressure deletes the inconvenient ones.

A second discipline follows: characterize narrowly. Pinning the output of a whole report, including its whitespace and its column widths, creates a test that fails whenever anyone touches formatting. Pinning the parsed records rather than the rendered text keeps the test attached to behaviour the consumer depends on.

Golden master when the output is too wide

Unit-level characterization works when the observable output is a handful of values. For a report generator, a pricing engine, or a parser whose output is a large document, writing assertions by hand is impractical, and the golden-master technique replaces the hand-written oracle with a recorded artefact:

import hashlib, json, pathlib, subprocess

GOLDEN = pathlib.Path("tests/golden")

def run_case(name: str, payload: dict) -> str:
    out = subprocess.run(
        ["legacy-report", "--json"],
        input=json.dumps(payload).encode(),
        capture_output=True, check=True,
    ).stdout
    # Normalise before hashing: line endings and trailing space are not behaviour.
    normalised = out.decode("utf-8").replace("\r\n", "\n").rstrip()
    digest = hashlib.sha256(normalised.encode()).hexdigest()
    expected = (GOLDEN / f"{name}.sha256").read_text().strip()
    assert digest == expected, f"{name}: output differs from recorded behaviour"
    return digest

The artefact can be a hash, as above, or the full output stored for diffing; the full output is more useful when the test fails, because the diff shows what changed. Hashing saves storage and forces normalisation to be explicit, which is usually where the real bugs hide. Note the normalisation step: any transformation applied before comparison is a decision about what counts as behaviour, and it must be reviewed as carefully as the assertions would be.

Golden masters scale to output nobody can read, which is their advantage. Their disadvantage is that they compare everything, including the parts nobody cares about, and a single changed timestamp turns the test red for reasons unrelated to the change under review.

Reviewing a recorded artefact is a different act from reviewing a test. A changed approval file produces a diff of the output, and the reviewer has to read it as a specification change rather than as a test edit. The practice that keeps this honest is to require the diff to be accepted explicitly, with the recorded output committed alongside the code change that produced it, so that the history shows both. A suite where approval files can be regenerated by a single command without inspection provides no protection at all: the regeneration erases the evidence that anything changed.

Fixtures need the same care. A characterization test that depends on a shared database whose contents change between runs is not deterministic, and the usual repair is to build the fixture inside the test from explicit rows. Where that is too slow, a seeded fixture restored from a snapshot gives each run the same starting state.

Non-determinism is the main obstacle

Most legacy units are not deterministic. Sources of variation in observed output include wall-clock timestamps, generated identifiers, hash-map iteration order in languages where it is unspecified, locale-dependent formatting, floating-point aggregation order, and concurrent completion order.

Each has a standard treatment. Time is injected through a seam and frozen. Identifiers are generated from a seeded sequence or replaced with a placeholder during normalisation. Iteration order is fixed by sorting before serialisation. Locale is pinned explicitly in the test environment, because the process default depends on environment variables the test does not control. Floating-point results are compared with a stated tolerance, and the tolerance is a decision: comparing text forces bit-exactness, which is achievable but brittle across compiler versions and optimisation levels, while comparing with a tolerance admits changes that a downstream rounding rule may not tolerate.

The worst case is genuine concurrency. A unit whose output depends on which of two threads finishes first cannot be pinned to one output. It can be pinned to an invariant: the set of records is the same regardless of order, which is a weaker guarantee but an honest one.

What the suite does and does not establish

A passing characterization suite establishes that the code’s observed outputs are unchanged for the recorded inputs. It establishes nothing about inputs not recorded, and the temptation is to read coverage numbers as evidence. Line coverage says which lines executed; it does not say that their effects were observed. A function that writes to a table and returns nothing can have full line coverage and zero behavioural assertions, because the only observable output is the table, and nobody asserted on it.

The suite also says nothing about whether the behaviour is right, which is why it is the wrong tool for judging a replacement. It is the right tool for detecting that a refactor preserved behaviour, and the foundation for the stronger comparison in differential testing between old and new implementations, where both systems are executed on the same generated inputs. Characterization tests answer “did this change”; differential tests answer “do these two agree”, and agreement across a broader input space is what a write-path cutover eventually relies on.