Coverage Criteria and What They Cannot See

Coverage tools report a percentage, and the percentage is almost always read as a quality measure. It is not one. It is a measure of how much of the program the suite executed, which is a necessary condition for testing that code and nowhere near a sufficient one. A suite can reach 100 percent statement coverage while asserting nothing, and a suite can be thorough at 70 percent.

The criteria are still useful, because each one is a precise statement about a set of paths, and precision is what makes a coverage gap actionable. The work is in knowing which set each criterion quantifies over, and which defects fall outside it.

Criteria as adequacy requirements

A coverage criterion defines a set of test requirements — elements of the program that must be exercised — and a suite satisfies the criterion when every requirement is met by at least one test. The definition is worth taking literally, because it makes the weaknesses obvious.

Statement coverage requires every executable statement to run. The requirement set is the set of statements. A suite satisfying it executes every line at least once, and may take a single path through a function whose branches each contain one statement.

Branch coverage (also called decision coverage) requires every edge of the control-flow graph to be taken: both outcomes of every if, every case of a switch that has a statement, every loop body and every loop exit. Branch coverage subsumes statement coverage except for unreachable code and statements after a non-returning call.

Condition coverage requires every atomic boolean subexpression to evaluate both true and false. For if (a && b), condition coverage requires a true once and false once, and b true once and false once — but not necessarily in the same test, so (true, true) and (false, false) satisfies condition coverage while never exercising the a=true, b=false case where short-circuit evaluation skips b.

Multiple condition coverage requires every combination of atomic conditions, which for n conditions is 2^n combinations per decision. It is the criterion that closes the short-circuit gap and the one that becomes infeasible as n grows; modified condition/decision coverage (MC/DC), used in avionics software standards, requires that each condition be shown to independently affect the decision outcome, which needs n + 1 cases rather than 2^n.

The subsumption order is real: MC/DC implies branch coverage, branch implies statement. Choosing a weaker criterion is choosing a smaller requirement set, not a different kind of guarantee.

Why the number is reachable without testing anything

Statement coverage counts execution. An assertion is not a statement the coverage tool requires, and a test that calls a function and discards the result covers every line inside it. Consider a suite whose tests all take the form “call the function; if it throws, the test fails”. Every line of the function is covered, and no output is ever checked. The report reads 100 percent.

This is not a hypothetical construction. It is what a suite looks like when tests are written to raise the number: the cheapest way to cover a line is to call it, and the cheapest call is one that asserts nothing. Coverage-driven development therefore has a predictable failure mode in which the metric improves while detection does not.

The second gap is the requirement. Coverage is computed against the code, so any behaviour the code does not implement is invisible. A specification with five validation rules, three of which were never written, has no lines to cover for the missing two. A suite at 100 percent branch coverage of a half-implemented feature is a suite that verifies the half.

The third gap is the input space. A branch can be covered by one input that happens to work and left untested for the inputs at its boundaries. Covering if (count > 0) with count = 1 says nothing about count = 0, which is the value the branch exists to handle. This is exactly the boundary analysis that partition and boundary design supplies, and it is orthogonal to structural coverage: one criterion selects inputs, the other reports paths.

Reading a coverage report usefully

A coverage report is most informative when read per file and per uncovered region rather than as a total. Three patterns in the per-file numbers carry signal.

Uncovered error handling is the most common and the most defensible: the branch that runs when the database is down, the catch block for a parse failure, the retry path. These are hard to reach, and the honest response is to reach them with a double or a fault injection rather than to accept them as untestable. An uncovered catch block is a block that has never run, and code that has never run is code that has never been checked.

Uncovered defensive branches are the second pattern: null checks, impossible-state assertions, default arms in exhaustive switches. Some are genuinely unreachable and should be deleted; some are reachable only through a caller the suite does not construct. Deleting the first kind reduces the maintenance surface, and reaching the second kind finds bugs.

The third pattern is a file at high coverage whose tests are all in one function. Line coverage aggregates over statements, so a large function with a single linear path reports high coverage from one test that walks it end to end. Per-function and per-branch views show the shape that the total hides.

# Threshold policy expressed as data: gate on the change, not on the total.
scope:      patch        # lines touched by the change
target:     80%          # below this, the change is rejected
project:    no-decrease  # the overall total may not fall
exclusions: generated/**, vendor/**

Gating on the patch — the lines a change touches — avoids two pathologies of a project-wide target. It does not force a rewrite of an old suite to reach a number, and it does not let a large new feature land with no tests because the total stayed above the threshold.

Data-flow coverage and the definition-use gap

Branch coverage is blind to a specific and common defect class: a variable assigned in one place and used in another, where the assignment is wrong for some path between them.

Data-flow criteria address it by treating definition-use pairs as the requirements. A definition is an assignment to a variable; a use is a read; a du-pair is a definition and a use reachable from it with no intervening redefinition. All-defs coverage requires each definition to reach at least one use. All-uses requires each du-pair to be exercised, which is strictly stronger: a variable reassigned on two branches and read afterwards has two du-pairs, and all-uses requires both paths to the read.

The classic defect all-uses catches and branch coverage misses is the swap:

void swap(int *a, int *b) {
    int t = *a;
    *a = *b;
    *b = *a;   /* should be t; branch coverage of this function is 100% */
}

The function has no branches, so branch coverage is trivially satisfied by one call. Data-flow analysis exposes the defect: the definition of t at the first line reaches no use, so all-defs is unsatisfied. A criterion that reasons about values rather than edges finds what a criterion that reasons about edges cannot.

All-uses is also expensive to satisfy, because du-pairs multiply with the number of paths between definition and use. In code with loops, the pair set is finite but large, and the test count needed to satisfy it can exceed what the suite can afford. The usual practice is to apply it selectively, to functions where state threading is the risk.

Mutation testing as a check on the criterion

The direct measure of whether a suite’s coverage means anything is whether the suite detects small deliberate changes to the code. Mutation testing generates such changes — flip a comparison operator, replace a constant, delete a statement — and counts how many survive the suite. A high survival rate at 100 percent coverage is proof that the coverage is decorative.

The two techniques are complementary in an exact way: coverage measures whether a requirement was reached, mutation measures whether reaching it was checked. A mutant that changes a branch condition is killed only by a test that reaches the branch and asserts on the difference. Coverage tells you which mutants are reachable; mutation tells you which were noticed. The mechanics and cost of the technique are taken up in mutation testing and test suite quality.

Where structural coverage stops being the right measure

Three situations make line-oriented coverage a poor fit, and in each the criterion is measuring an artefact rather than behaviour.

Generated or vendored code. Coverage over code that no one wrote is a number about a dependency. Exclude it, and be explicit that the exclusion is a scoping decision rather than an improvement.

Declarative and configuration surfaces. A schema, a template, a rule table: the branches live in an interpreter the suite does not own, and the interesting cases are the data. Covering the data means enumerating partitions of the configuration space, which is the input-design problem again.

Concurrency. Branch coverage of a function with a lock says nothing about the interleavings. The control-flow graph collapses every possible schedule into one path, and the criteria are defined over that collapsed graph. Finding races requires either systematic exploration of schedules or stress execution, neither of which produces a coverage number. The reasons the second is unreliable are the sources of nondeterminism catalogued in nondeterministic tests and flaky suites.