Characterization Tests as a Refactoring Safety Net

A refactoring needs an oracle: a way to decide whether the new code produces the same observations as the old. When a specification exists, the oracle is the specification. When the code is ten years old and the people who wrote it have moved on, no specification exists, and the only available oracle is the behavior of the current implementation. A characterization test records that behavior so it can be compared later.

The name is precise. The test does not characterize what the code should do. It characterizes what it does, including the parts that are obviously wrong.

Capturing behavior that nobody specified

The procedure is mechanical and deliberately uncreative. Pick a call into the system under test, run it, observe the result, and write a test asserting exactly that result. Repeat until the assertions cover the paths that the planned change will touch.

The uncreativity is the point. The instinct when writing a normal test is to reason about correct output and to write the assertion that a correct implementation would satisfy. That instinct is fatal here, because it encodes a guess about intent, and the guess will disagree with the implementation in exactly the cases where the change is most likely to break something.

A typical characterization test for a date formatter with an undocumented quirk:

from datetime import date

def test_quarter_end_formatting():
    # No claim about whether this is right; it is what the code returns.
    assert format_period(date(2024, 3, 31)) == "2024-Q1-13"
    assert format_period(date(2024, 4, 1)) == "2024-Q2-01"
    assert format_period(date(2024, 12, 31)) == "2024-Q4-53"

The 13 and 53 in the expected values are almost certainly a week-numbering bug. The test asserts them anyway, because the purpose of the net is to detect change, not to enforce correctness. If the refactoring accidentally fixes the bug, the test fails, and the failure is informative: someone now knows that the fix is a behavior change requiring its own decision, separate from the structural work.

Where to cut the seam

A characterization test needs a boundary at which observations can be captured. Code with no testable entry point has to be given one first, and creating that seam is itself a change to the code, made before any refactoring. The usual options are extracting a function so it can be called directly, replacing a direct dependency on a clock or a network client with an injectable one, and reducing a wide interface to the narrow slice the test needs.

That preparatory work is the part practitioners underestimate. In code with heavy static coupling, adding the first seam may require more edits than the refactoring that follows. It is still worth doing first, because a seam added after the rewrite cannot be used to check the rewrite.

Golden-master comparison

For output that is large, structured, or hard to assert piecewise, the assertion is a comparison against a stored file. The test writes the current output to a file once, that file is committed, and every subsequent run diffs against it.

import json, pathlib

GOLDEN = pathlib.Path("tests/golden/invoice_summary.json")

def test_invoice_summary_matches_golden():
    actual = render_summary(SAMPLE_INVOICES)
    if not GOLDEN.exists():
        GOLDEN.write_text(json.dumps(actual, indent=2, sort_keys=True))
        return                                    # first run records the baseline
    assert actual == json.loads(GOLDEN.read_text())

Two details make this work rather than annoy. Serialization must be deterministic, so keys are sorted and any timestamp in the output is injected rather than read from the clock. And the diff must be readable, because a failing golden test is useful only if the person reading it can see which field moved. A binary golden file produces a failure with no information in it.

The same technique applies at a coarser grain: capture the requests a service makes to its dependencies and replay them. The captured traffic is a specification of the interaction, and a refactored service that issues different requests fails the comparison even when its own responses are identical.

Coverage of the observation set, not of the lines

Line coverage answers the wrong question for this purpose. A characterization suite’s value is the fraction of the observation set it samples, and line coverage is a proxy that can be high while the sample is useless.

A branch that runs but whose result is discarded is covered and unobserved. A test that calls a function and asserts only that it does not throw covers the line and characterizes nothing about its return value. The metric that matters is how many distinct behaviors the assertions distinguish.

Three heuristics raise the sampled fraction cheaply. Enumerate the input space at its boundaries rather than at random interior points: empty, one element, maximum size, and the values adjacent to any threshold in the code. Record the exception type and message, not just the fact that a call failed, since the type is part of the contract. Capture side effects, which for a function that writes to a database means asserting on the resulting rows rather than on the return value.

Mutation testing is the direct measurement of the sample. Introduce a small change to the code, run the suite, and see whether it fails. A surviving mutant is a behavior the suite does not observe. Running a mutation tool over a module before refactoring it gives a concrete list of the unobserved behaviors, which is a better use of time than reading coverage reports.

What the net cannot hold

A characterization suite is a sample, and every sample has blind spots that are structural rather than accidental.

Concurrency. Tests run one schedule. A characterization suite over concurrent code records the results of the schedules it happened to execute, and a refactoring that changes locking can preserve all of them while introducing a new interleaving that fails. No amount of test volume fixes this, because the space is exponential in the number of operations.

Resource behavior. The suite observes return values, not descriptor counts or peak memory. A refactoring that leaks a handle per call passes every test and fails after enough calls.

Timing. Any test with a timeout is a timing assertion with an unspecified threshold. It characterizes that the operation is “fast enough for this machine today,” which is not a property that can be preserved deliberately.

Inputs nobody enumerated. This is the ordinary sampling gap and the largest source of post-refactoring defects. The suite covers what someone thought of; production covers the rest.

The oracle’s own bugs. If the baseline was recorded from a buggy implementation, the test enshrines the bug. That is the intended behavior and it still means the suite cannot be used to argue that the code is correct, only that it is unchanged.

# A mutant that survives is a gap, not a pass.
# assert_equivalent(before, after, cases) is only as strong as `cases`.

The practical consequence is that a green characterization suite licenses the refactoring, not the release. The two decisions are separate and have different evidence requirements.

Sequence relative to the change

Order matters more than the technique. The suite must exist, must pass against unmodified code, and must be committed before the structural edit begins. A test written after the change encodes whatever the changed code does, so it confirms nothing.

Two rules follow. Every test must fail when the old behavior is removed; a test that passes against both the old and a deliberately broken implementation is not measuring anything. And the suite must run fast enough to be run constantly, because its value comes from the number of times it is consulted during the edit, not from the thoroughness of a single run.

Once the net exists, the refactoring can proceed in small steps with a check after each one. That is the same discipline that makes Extract Function and the Preconditions the Compiler Checks safe: the edit is small, the net is fast, and a failure is attributable to one change. For a large rewrite, the net is also what makes Expand, Migrate, Contract: Changing an Interface Without a Flag Day verifiable, since the two live paths can be compared against the same recorded behavior before the old one is deleted.