Mutation Testing and What Coverage Cannot Tell You

A function that computes a total is called by a test that checks nothing about the result. The test executes every line, so coverage reports the function as fully covered, and the report is accurate: the code ran. Whether the suite would notice if the function returned a wrong number is a different question, and it is the question mutation testing asks by changing the program and re-running the tests. The measurement it produces is not a percentage of code executed but a percentage of deliberate faults detected, and the two numbers disagree on exactly the suites that most need attention.

Detect instead of execute

A mutant is a copy of the program with one small syntactic modification: a relational operator swapped, a constant changed, a statement deleted, a conditional negated. The suite runs against the mutant. If some test fails, the mutant is killed, meaning the suite distinguished the mutated program from the original. If every test passes, the mutant survives: there is a behaviour change that no assertion in the suite observes.

The mutation score is the fraction of non-equivalent mutants killed. The appeal of the metric is that a surviving mutant is a fact with a location — this line, this operator, not detected — rather than a statistic. A developer reading the list can see immediately whether the survivor reveals a genuinely untested behaviour or is an artefact, and can often fix it with one assertion. Coverage, by contrast, reports a line as covered and says nothing about whether any test would have noticed it misbehaving; the mutation score is the measurement that distinguishes those two situations, which is why the report from How Coverage Tools Count Executed Lines and Branches and the survivor list are worth reading together.

Operators, sites, and the size of the search space

The modification rules are called mutation operators, and they are chosen to mimic small mistakes rather than arbitrary changes. Common families include replacing one relational operator with another (< for <=, == for !=), swapping arithmetic operators, replacing a constant with zero, one, or a boundary neighbour, negating a boolean condition, removing a statement, removing a method call with no used return value, and replacing a returned value with a default. The set matters because it defines what the metric can detect: a suite that would notice a wrong comparison but not a missing side effect scores well against a boundary-only operator set and badly against a statement-deletion one.

Sites are the places an operator can be applied, and the count grows quickly. Every comparison contributes at least two mutants, one for each neighbour in the operator lattice; every condition contributes a negation mutant; every constant contributes replacement mutants. The full operator set applied to every site commonly produces more mutants than the program has executable lines, which is what makes the unoptimised form of the technique expensive: each mutant requires a test run, and the mutants are independent.

The total cost is therefore the number of mutants multiplied by the cost of running the tests that must be run to decide each one. Two techniques attack the product, not the factors, and both are standard. Higher-order mutation generates a single program containing many mutations and uses a driver to activate them one at a time, so compilation and startup are paid once; a related technique copies the original and injects the mutation at run time. Coverage-directed execution uses the per-test coverage trace to run only tests that executed the mutated line, which is where mutation testing and test selection draw on the same data structure — the same inverted index from location to tests that Change-Based Test Selection and Where It Stops Being Sound uses to decide what to run. Since a mutant of a line covered by three tests needs only those three to be tried, the product is reduced by the ratio of coverage to suite size, which is large in a suite with many unrelated areas.

import ast

SWAPS = {ast.Lt: ast.LtE, ast.LtE: ast.Lt, ast.Gt: ast.GtE, ast.GtE: ast.Gt}

def compares(node):
    """Depth-first pre-order, the order tree visitors use."""
    for child in ast.iter_child_nodes(node):
        if isinstance(child, ast.Compare):
            yield child
        yield from compares(child)

def mutants(source):
    sites = [n for n in compares(ast.parse(source))
             if any(type(op) in SWAPS for op in n.ops)]
    for k in range(len(sites)):
        tree = ast.parse(source)                  # each mutant is a fresh copy
        node = list(compares(tree))[k]
        for i, op in enumerate(node.ops):
            if type(op) in SWAPS:
                node.ops[i] = SWAPS[type(op)]()   # exactly one swap per mutant
                break
        yield ast.unparse(tree)

The sketch above mutates one comparison operator per generated mutant; a real tool enumerates every applicable operator at every site and hands each mutant to a runner that compiles it, executes the tests that cover the mutated line, and classifies the outcome as killed or surviving. Parsing the source again for each mutant is what guarantees the single-difference property: transforming a tree that has already been mutated once would produce a mutant differing from the original in two places, which the score would then attribute to a single operator.

Equivalent mutants and an undecidable denominator

Some mutants change the program’s text without changing its behaviour. A statement that computes a value never read can be deleted harmlessly; a comparison whose two branches do the same thing can be flipped; an arithmetic rewrite can be valid given a precondition the programmer knows holds but which is not asserted anywhere. Such a mutant can never be killed, because there is nothing to detect — the two programs are extensionally equal on every input.

Detecting equivalence is undecidable in general, since it requires proving the two programs compute the same function. Practical tools therefore leave equivalent mutants in the denominator, where they depress the score, and developers either mark them as suppressed after inspecting them or accept a ceiling below one. The consequence for interpretation is that a mutation score of 100% is not earnable in general and near-100% is a statement about the suppression list as much as about the tests. It also explains why the metric is best used per change and per class rather than as an organisation-wide target: a single equivalent mutant in a small module swings the percentage far more than in a large one.

A second class of unkillable mutant resembles the first but arises from reachability. A mutation inside a branch guarded by an earlier condition may make the branch unreachable rather than equivalent; no test can kill it because no input reaches it. Coverage-directed generation avoids most of these by mutating only lines that some test has executed, which is an argument for using coverage data to decide where to mutate even when the reported metric is mutation score.

Reading the survivors

The list of surviving mutants is more useful than the score, and it divides into recognisable categories. A survivor on an input validation boundary usually means the tests exercise the valid path and never the invalid one. A survivor on a return value that no assertion inspects points at a test that executes a call for its side effects but verifies nothing — the case where the coverage number is highest and the suite’s checking is weakest. A cluster of survivors in one function is a signal that the function’s tests were written against the implementation rather than the specification, because they exercise the code paths without pinning the outcomes. A survivor caused by deleting a state-changing call indicates that the tests set up state and check a later result, but never check the mutation itself.

There is also a category that is noise rather than signal: mutants in generated code, in logging or serialisation that only produce side effects, and in code whose behaviour is observed only by a human reading output. Suppressing whole classes of site, rather than individual mutants, keeps the report focused, and the suppression is a decision worth recording because it changes what the metric can detect.

Two properties of the technique are worth stating plainly, because they bound what it means. Mutation testing measures the tests, not the program: a high score says the assertions are sensitive to small changes, and says nothing about whether the specification was implemented correctly. And it can only mutate code that exists, so a requirement that was never implemented, or a branch that was forgotten, produces no mutants at all — the gap is invisible to this metric for the same reason it is invisible to coverage. Mutation testing narrows the gap left by coverage, from “the line ran” to “the line’s behaviour was checked”, and stops there.

The practical deployments reflect that: mutation runs are typically applied to the code a change touches rather than the whole tree, with the score for the changed lines reported alongside their differential coverage, and with the survivors reviewed the way a code review is. Both numbers are cheap to misunderstand and hard to fake, which is the whole basis for using them together. A suite whose tests were written to execute code accumulates coverage and leaves survivors; a suite whose tests were written to constrain behaviour shows the opposite pattern, and the pair of reports is what makes the difference visible.