Test Oracles and Where Expected Values Come From

Generating a test input is a mechanical problem with mechanical solutions: enumerate partitions, pick boundaries, randomize, fuzz. Deciding whether the program’s response to that input is correct is a different problem, and it is the one that limits what any suite can detect. A test without a reliable verdict is a program that runs code and asserts that nothing exploded. It catches crashes and exceptions, and it is silent about wrong answers.

The term for the mechanism that produces a verdict is an oracle. Every assertion in every suite embeds one, whether or not the author thought about it, and the properties of the oracle bound the defects the suite can find.

The asymmetry between input and verdict

For a function f(x) = y, a test needs three things: an x, a way to invoke f, and a way to know the right y. The first two are cheap. The third is the problem, because the most obvious oracle for f is a second implementation of f, and a second implementation must itself be trusted.

Some functions admit cheap oracles. sort is correct exactly when the output is a permutation of the input and adjacent elements are ordered; both checks are linear and independent of any sorting implementation. A parser paired with a printer admits the round-trip oracle: parse then print then parse again must yield the same tree. A serializer with a schema admits a structural oracle: the output must validate against the schema. None of these require a reference answer.

Many functions admit nothing of the kind. A compiler’s correct output is machine code that computes the specified function; no cheap predicate decides that. A ranking function’s correct output is a particular order, which only the product requirement defines. A pricing engine’s correct output is a number that depends on tax law. These are the cases where the oracle problem bites, and where the practical techniques are all approximations with known gaps.

Specified values, and what they pin down

The simplest oracle is a literal expected value written next to the input: assert f(3) == 14. It is also the most common and the most dangerous, for a specific reason. The literal has to come from somewhere — a hand computation, the specification, or the implementation’s own output captured once and pasted in. The third source is indistinguishable from the first two at review time and turns the test into a snapshot of behaviour rather than a check of it.

A literal that was computed from the specification is a genuine oracle, and it is worth stating why it is trustworthy: it was derived independently of the code under test. A literal captured from a run of that code has no such independence and encodes whatever the code did, defects included. The distinction is invisible in the diff, which is why suites accumulate assertions that lock in wrong values.

The related trap is asserting on more than the contract. A report function that returns a formatted string and a list of rows has a specified contract for the rows and an incidental one for the string. Asserting the exact string couples the test to column widths and separators that the specification never fixed. When the format changes legitimately, the test fails for no defect, and the failure teaches the team to update expectations reflexively — which is how an oracle stops being one. Assert on the rows, and on the parts of the string the specification names.

Golden files and the review they require

For outputs that are large, structural, or awkward to express inline — an HTML rendering, a generated schema, a compiler’s assembly listing — a golden file stores the expected output beside the test. Comparison is exact or normalized, and regeneration is a single command.

The economics are what make golden files work or fail. Regeneration must be easy, because it will be needed; and the diff must be reviewed, because regeneration without review converts the suite into a change detector with no notion of correctness. A golden file that is regenerated whenever it fails and merged without reading is strictly worse than no test, since it consumes review attention while providing no signal.

Normalization decides how much the golden file measures. Timestamps, addresses, iteration order of hash containers, and absolute paths must be scrubbed, or every run differs. Over-normalizing has the opposite cost: strip enough of the output and the file stops distinguishing correct from incorrect. The honest form is to normalize only the fields the specification leaves unconstrained.

Metamorphic oracles: checking relations instead of values

When no expected value is available, a relation between inputs and outputs often is. Metamorphic testing exploits such relations: run the program on a source input, transform the input in a way whose effect on the output is known, run again, and check that the effect appeared.

Concrete relations for common shapes:

• Permutation invariance. For a function that returns a set or a sorted collection, permuting the input must not change the output. For a function that returns an aggregate, splitting the input into two disjoint parts and combining the results must equal the result on the whole.
• Monotonicity. A comparator-based ranking must preserve order under a monotone transformation of the score. A threshold classifier must not flip its decision when the input moves further from the boundary in the same direction.
• Scaling. For a linear pricing rule, doubling the quantity must double the pre-discount total. For a unit-conversion function, applying the conversion twice in opposite directions must return the original value within a stated tolerance.
• Idempotence. Normalizing an already-normalized value must change nothing. Applying a deduplicating filter twice must not change the result.

The strength of a metamorphic oracle equals the strength of the relation. Permutation invariance of a sort catches any order-dependent bug and misses any bug that produces the same wrong answer for every permutation — a comparator that always returns negative passes it. Relations compose, so a handful of independent ones narrow the space considerably, but none of them is a proof.

The practical advantage is reach. Metamorphic checks cost one extra execution per case and apply to inputs generated in bulk, which makes them the natural oracle for randomized input generation, as described in property-based testing.

Differential testing against a second implementation

Two independent implementations of the same specification, fed the same inputs, must agree. This is back-to-back testing, and it is how most compiler and interpreter conformance work is done: a new implementation is compared against an established one over a large generated corpus.

The property that makes it work is independence of failure. Two implementations written from the same misreading of the specification agree with each other and disagree with the specification, and the differential test reports success. Differential testing finds implementation bugs, not specification bugs, and it finds them only where the implementations genuinely diverge — shared libraries, shared algorithms, or a common ancestor defeat it entirely.

The disagreement is also only a pointer. A differential failure says one of the two is wrong, not which; adjudicating it requires reading the specification, which is the work the technique was supposed to avoid. Mismatches on undefined behaviour — integer overflow, evaluation order, uninitialized reads — are common and are not defects in either implementation.

A useful variant compares an optimized path against an unoptimized one within the same codebase: a vectorized kernel against its scalar reference, a cached path against a cold one. Here the two sides share a specification by construction, so any disagreement is a real defect, and the check can run on every input the fast path sees.

Oracles as a property of the whole suite

The suite-level view is that oracle strength is a resource distributed unevenly, and the distribution is usually wrong. Tests with cheap oracles — no exception thrown, non-null returned, status code 200 — are written in bulk because they are easy, and they are weak. Tests with specified values are written for the two or three cases someone cared about, and they are strong. Coverage metrics count both equally, which is why coverage alone is a poor guide to suite quality; the interaction between coverage criteria and oracle strength is examined in coverage criteria and blind spots.

A practical audit asks, for each test, what defect it would catch and what defect of the same family it would miss. A test asserting only that a parse succeeds catches crashes and misses every semantic error in the parsed result. A test asserting three fields of a five-field result catches errors in those three. Writing the answer down for a sample of the suite usually identifies the region where coverage is high and verdicts are absent.

// Weak: any non-empty array passes, including one built from the wrong records.
const rows = loadReport(source);
expect(rows.length).toBeGreaterThan(0);

// Strong: the oracle is a relation, independent of the loader's internals.
const rows2 = loadReport(source);
expect(new Set(rows2.map(r => r.id)).size).toBe(rows2.length);   // ids unique
expect(rows2.map(r => r.total)).toEqual(rows.map(r => r.total)); // stable across runs
expect(rows2.reduce((a, r) => a + r.total, 0)).toBeCloseTo(expectedSum, 2);

The second block is not much longer, and it can fail for four distinct reasons. That is the shape of a useful oracle: it constrains the output along axes the specification fixes, and it leaves the rest alone.

Where no oracle exists

Some outputs are genuinely unverifiable by the suite that produces them: the correctness of a floating-point library’s last bit, the equivalence of two compiler optimization levels for undefined programs, the visual fidelity of a renderer. The honest responses are to shrink the claim — verify the property that is checkable and state the rest as an assumption — or to move the verification to a different mechanism entirely, such as a proof assistant or a reference implementation validated by other means.

The dishonest response, and the common one, is an assertion that passes for the wrong reason: a snapshot of the current output, a tolerance wide enough to admit any answer, a check on a field that cannot vary. Those assertions are indistinguishable from real ones in a coverage report, and the distinction only surfaces when a defect slips through a test that was supposed to catch it.