How Coverage Tools Count Executed Lines and Branches

A coverage report looks like a measurement: 84% of lines, 71% of branches. It is closer to a quotient whose numerator and denominator were both chosen by a tool. The numerator is the set of regions some instrumentation observed, the denominator is the set of regions that same tool decided were coverable, and neither is a property of the program. Understanding which mechanism produced the events explains why two tools disagree about the same run and why a number that rises is not automatically an improvement.

What instrumented execution records

There are two families of mechanism. The first asks the runtime to notify a callback each time control reaches a location: an interpreter exposes a trace hook that fires per line, per function call, and per return, and the hook records the location in a set. Python’s sys.settrace is the long-standing interface, and the interpreter has since added a finer-grained monitoring interface aimed at exactly this use case, because a per-event callback implemented in the interpreter’s own language is expensive relative to the work being measured.

The second family moves the work to compile time. The compiler inserts counters into the generated code, one per basic block or edge, and emits a separate table describing which source regions those counters correspond to. At runtime the program only increments integers; at exit, the counters are written to a data file. Gcov works this way, with a notes file produced at compile time describing the instrumentation and a data file produced by the run that holds the counts. LLVM’s source-based coverage uses the same split, writing a raw profile at run time and combining it with a mapping emitted into the binary, then merging and reporting offline.

The structural difference is where the cost lands. A callback-based tracer pays a dispatch cost per executed event, which is why instrumented runs of an interpreted program are much slower than uninstrumented ones and why the slowdown grows with the number of events a line of source produces. A counter-based instrumenter pays a few instructions per basic block, plus the memory traffic of updating the counters, which can itself perturb caches. In both cases the measurement changes the thing measured: timing tests and coverage runs are mutually exclusive, and a coverage run of a suite with deadline assertions tends to produce failures that have nothing to do with the code.

/* Counter-based instrumentation, roughly what a compiler emits:
   one counter per basic block, incremented on entry. */
static unsigned long counters[64];
void f(int x) {
    counters[0]++;              /* function entry block */
    if (x > 0) {
        counters[1]++;          /* then-block */
    } else {
        counters[2]++;          /* else-block */
    }
    counters[3]++;              /* join block */
}

In the snippet, four counters describe a function with two branches. Zero in counters[2] means the else path never ran, which is information a line-level view cannot express, because both blocks belong to a line that executed.

Four denominators that are not interchangeable

A tool that reports a percentage is dividing by some set of regions, and the choices differ.

Line coverage counts source lines that contain executable code. The hard part is deciding which lines those are: a line containing only a closing brace has nothing to execute, and a line with several statements on it is either covered or not as a unit. Line coverage is the easiest to read and the easiest to game, because one long line can absorb many statements.

Statement coverage counts executable statements, which is finer, and requires the same analysis a compiler’s front end already does for basic-block construction. Two statements on one line are counted separately, which makes the metric behave sensibly on minified or generated source.

Branch or edge coverage counts the outgoing paths of decision points, so an if contributes two entries. This is the metric that distinguishes a condition tested one way from a condition tested both ways. A partially covered branch is a specific, useful report: the tool knows the decision was evaluated and knows only one successor block was reached. Relational operators complicate it, because a comparison that is false for every tested value looks like a covered edge, and a short-circuiting boolean sequence has intermediate decision points that may never be evaluated at all.

Region coverage generalises all of the above: the compiler partitions the function into contiguous source regions, each with a count, and the report can colour any span of the source. LLVM’s source-based instrumentation is region-based, which is why it can mark an expression’s subterms rather than a whole line.

Comparing numbers across tools is therefore a category error unless the denominator matches. A report of 92% lines and another of 78% branches may describe the same execution exactly.

Merging data from many processes

Any test suite that spawns children or distributes work across workers produces several profile files for one logical run. Merging them is a sum over counters keyed by the location the counters describe, and it works because the counter layout is deterministic for a given binary. The two failure modes are both about identity.

The first is filename collision. If every process writes to the same path, they overwrite each other and the merged result silently reflects whichever process finished last. The convention in profile runtimes is a filename pattern containing a token that the runtime substitutes with the process id, so each process writes a distinct file and the collector globs them afterwards. For interpretive runtimes the equivalent is a data file per worker and an explicit combination step, plus the discipline of deleting stale files before a run: a leftover file from a previous run inflates the counts of whatever it covered.

The second is source identity. The mapping from counters back to source is keyed by the file path as it appeared at compile time, sometimes with a hash of the content to guard against staleness. A report generated against a modified source tree is either rejected or, worse, silently misaligned — counters attributed to the wrong lines because lines were inserted above them. Tools that embed a content hash can detect this and discard the data; a tool that stores only a path cannot.

Counter updates also have a thread-safety question. A plain increment is not atomic, so two threads reaching the same block concurrently can lose an update. Coverage counts are usually approximate by design, and a runtime may accept the loss rather than pay for atomic increments on the hot path, but a report that claims a line was never executed when it was executed is a correctness bug in the report rather than an acceptable approximation, so implementations generally guarantee at least a monotone “was reached” flag even when exact counts are best-effort.

Merging across a run, and across runs

The merging model extends to suites that run over days. Counters accumulate, so a build system can keep a baseline profile and add the day’s runs to it, which makes “has any test ever exercised this line” answerable without re-running everything. That is the honest use of cumulative coverage: as a map of which code has no test at all, it is precise, and it is what makes a line list rather than a percentage the right interface for large codebases.

The corresponding measurement for a single change is differential coverage: of the lines the change touched, which were executed by the tests that ran with it. That metric has a smaller denominator, is unaffected by historical dead code, and is far more actionable than the global figure — a new branch with no test is a fact about the change, whereas the gap between 84% and 86% is usually a fact about which files happen to be in the report. It also pairs naturally with selecting tests by which code they exercise, the subject of Change-Based Test Selection and Where It Stops Being Sound, because the two share the same dependency map and suffer from the same blind spot when that map is incomplete.

What the number cannot say

Coverage is a statement about execution, not about checking. A test with no assertions that calls a function covers it completely, and a test that compares a result against itself covers an entire computation while checking nothing. Every mechanism above records that control reached a location; none records whether anything was asserted about the result, and none records whether the assertions that exist would fail if the code were wrong. That gap is not a defect of any particular tool but a limit of the metric, and it is the gap that Mutation Testing and What Coverage Cannot Tell You tries to close by measuring something different. Because the two metrics disagree, they are worth reporting side by side: coverage says the code ran, mutation score says the suite would notice if the code changed, and neither substitutes for the other.