How Assertion Libraries Recover Operands From a Failed Comparison

A failed comparison is the most common thing a test reports, and the least informative by default. Evaluating a == b produces a single bit, and the language’s own assertion machinery usually propagates that bit with no record of what the operands were. The message a developer sees at that point is the entire value the test suite delivered, so reconstructing the operands is a problem every test framework has to solve, and each of the four solutions in common use buys readability with a different currency.

An assertion knows a boolean, not a story

The semantics of an assert statement in a compiled language are defined by the language and are minimal. In C, the standard specifies that if the expression is false, the failure is reported with the text of the expression, the source file name, and the line number; it says nothing about the values. In Python, a bare assert statement raises AssertionError with the message argument if one was supplied, and with no message otherwise; nothing is captured about the operands.

The information needed for a good message is destroyed at the moment the comparison is evaluated. To get it back, something has to either (a) capture the text of the expression before evaluation, (b) capture the values of subexpressions during evaluation, or (c) hand the values to an assertion routine whose parameters are the operands. Each of those is a real technique in a widely used tool, and the differences between them are structural rather than cosmetic.

A fourth possibility, reading the failing frame’s local variables after the exception occurs, is the one that appears attractive and rarely works. Local variable tables are optional in many compilers, they are discarded under optimisation, and a frame cannot tell which of its locals participated in the expression. It can suggest candidates, not recovery.

Stringification happens before compilation

The C assert macro is the oldest mechanism and the cheapest. Its argument is text, and the preprocessor can turn text into a string literal with the # operator, so an expansion along the lines of

#define assert(expr)                                                     \
  ((expr) ? (void)0                                                      \
          : __assert_fail(#expr, __FILE__, __LINE__, __func__))

gives the failure handler the source text, the file, the line, and the enclosing function name without any runtime cost on the success path beyond the branch itself. The string is a compile-time constant, so it lives in read-only data and costs nothing to construct. glibc’s implementation reports exactly those four items.

This is why C assertion output reads as an expression rather than a value: assert(len < cap) reports len < cap failing, and the developer infers that len reached cap. The mechanism cannot do better, because the macro never sees the operands — it only sees source characters. Producing values requires the author to pass them, which is why test frameworks in C-derived languages tend to define their own macros with variadic arguments rather than rely on the standard one. The other cost is that assert is specified to be compiled out when NDEBUG is defined, so an assertion that has side effects in its expression behaves differently in the two configurations — a specification-level trap rather than an implementation quirk.

Rewriting the source to keep intermediate values

pytest takes the opposite approach: it changes the code before it runs. Test modules are not imported by the normal import machinery. A module finder installed by the runner intercepts the import, parses the source with the ast module, and rewrites every assert statement into code that evaluates the expression, keeps references to the subexpressions, and calls a helper on failure with those references. The rewritten module is then compiled and executed, and the compiled form is cached so the cost is paid once per source change.

import ast

class RewriteAssert(ast.NodeTransformer):
    def visit_Assert(self, node):
        self.generic_visit(node)
        # Keep the original test for control flow, but hand the operands to
        # the reporter so both sides can be rendered on failure.
        return ast.If(
            test=ast.UnaryOp(op=ast.Not(), operand=node.test),
            body=[ast.Raise(exc=ast.Call(
                func=ast.Name(id='_failure', ctx=ast.Load()),
                args=[node.test, ast.Constant(value=node.msg)],
                keywords=[]), cause=None)],
            orelse=[],
        )

The transformer above is a sketch, not what the real implementation does; the important part is the shape of the trade. Rewriting can recover values for arbitrarily nested expressions — both sides of a comparison, the tuple of a membership test, the operands of a boolean operator with the failing branch marked — and it can produce a structured explanation of the operator rather than a rendering of the source. The costs are that the code being run is not the code in the file, so traceback line numbers, coverage mapping, and debugger stepping all have to be reconciled with the original source; that only modules the rewriter claims are affected; and that the whole mechanism is defeated if the interpreter is started with optimisation enabled, because the assert statement is compiled away before any rewriter sees it. The same flag that removes the C macro’s effect at NDEBUG removes Python’s at -O.

Because rewriting is scoped to imported test modules, an assertion inside a helper imported from a non-test package is not rewritten. The failure still reports the values of the comparison if the helper itself is rewritten, and reports nothing if it is not. That asymmetry is the most frequent source of “why is this message so bad” questions.

Passing the operands explicitly

The third mechanism abandons source inspection. assertEquals(expected, actual) receives both values as arguments, so rendering them is a matter of string conversion, and a good implementation adds a targeted diff for the types it understands. Java’s assertion libraries are built this way, and the price is paid in the API surface: one overload or one matcher per kind of comparison, and the loss of the expression text.

That loss is smaller than it sounds when the failure is a comparison of two values, and larger when it is a compound condition. assertThat(items).areUnique() can report which element repeated, because the matcher receives the collection; an expression like a in b and c has no natural split into operands, so such libraries encourage writing it as two assertions. The design constrains the writing style, which is a legitimate cost to name rather than a defect, and the constraint is preferable to the alternative where a test calls code and checks nothing at all, the failure mode that Mutation Testing and What Coverage Cannot Tell You is built to make visible.

Go sits in the same family. There is no assertion facility in the testing package at all: a test calls t.Errorf or t.Fatalf with a format string and the values to print, and the standard library does nothing to derive them. The advantage is that failure output is exactly what the author wrote; the disadvantage is that the author has to write it for every assertion, and miscounted argument lists are caught by the type checker rather than silently producing a useless message.

Diffs, truncation, and operands that lie

Once both operands are available, rendering them is the remaining work, and it is where the cost moves. Comparing two sequences for a message normally starts by locating the common prefix and suffix, since the interesting region is between them, and then runs a diff algorithm over the remainder. The classic algorithm is Myers’s, whose cost is the product of the total input length and the size of the edit script, so a dissimilar pair of long sequences is expensive to render — enough to be noticeable when a suite contains many such failures and each is reported eagerly rather than counted.

Two failure modes follow from the rendering step. The first is truncation: an operand that stringifies to megabytes has to be cut, and the cut policy determines whether the useful part survives. A representation that shows the first few hundred characters of a large structure hides precisely the element that differs. The second is operands whose equality is not a total function. A comparison that returns a non-boolean — an array-like object producing an element-wise result — raises a separate error about truth value rather than reporting a mismatch, and an equality method with side effects can make the reported values differ from the values that were compared. Nothing in the assertion machinery can detect that; the message is only as trustworthy as the operands’ own string and equality implementations.

The message is also what makes a failure reproducible. A runner that groups failures by their text, as described in Reducing Failing Inputs to a Minimal Counterexample, depends on that text being stable for the same cause and different for different causes, which in turn depends on whether operand rendering includes addresses, timestamps, or generated identifiers. Assertion output is not merely cosmetic; it is the key that failure-aggregation systems index on.