How Doctest Decides Whether a Documented Example Passed

Every documentation set contains examples, and an example is a claim about behaviour. Most such claims are never checked: a code block is rendered, a reader copies it, and the first sign of divergence is a bug report. The doctest format takes the opposite position by making the example a complete observation, with input and output written as they would appear at a Python prompt, and then running it.

The format is a derivation of the interactive transcript. Lines beginning with >>> are the input; the lines that follow, up to a blank line or the next prompt, are the expected output. Whatever the interpreter writes to standard output while the input executes is the actual output. A single string comparison between the two decides the result. Everything else in the system exists to make that comparison survive the incidental variation in real programs.

def normalise(name: str) -> str:
    """Collapse internal whitespace and strip the ends.

    >>> normalise('  Ada   Lovelace  ')
    'Ada Lovelace'
    >>> normalise('   ')
    ''
    """
    return ' '.join(name.split())

Run under python -m doctest with verbose output, or collected by doctest.testmod(), this produces two comparisons and reports failures with the expected and actual values. The mechanism is small. The interesting part is what counts as equality.

The expected block is a byte-exact transcript

Captured output and the text in the docstring are compared as strings, and by default the comparison is exact, including trailing spaces on each line and the number of blank lines between them. This is a deliberate choice in favour of unambiguous examples: a reader who copies the block gets exactly the behaviour shown, with no interpretive latitude.

Exactness has two immediate consequences. The first is that a blank line terminates the expected output, so a program that legitimately prints an empty line needs the marker <BLANKLINE> in the expected block to represent it; without the marker the example ends early and the rest of the transcript is parsed as prose. The second is that the representation of a value, not the value itself, is what gets compared. The prompt compiles statements in single mode, where the value of an expression statement is echoed as its repr, which is why >>> 1 + 1 expects 2 while >>> print(1 + 1) also expects 2 but for a different reason: the first goes through the display hook, the second through standard output. Both reach the same redirected stream, so the checker sees identical bytes.

A repr is a function of the interpreter and the object, not of the concept being documented. A float example that expects 0.30000000000000004 documents binary floating point rather than the function under discussion, and the value is stable, so the example passes and teaches the wrong thing. An object whose default repr includes a memory address never passes twice in a row. Dictionary output is insertion-ordered in all supported Python 3 versions, so a dict example is stable today, but set iteration order is not specified for strings, and an example that prints a set is a latent failure that appears on a different run or a different hash seed.

Where the comparison is made looser, and what that costs

Four comparison options handle the common sources of incidental difference, and each one removes a class of failure by removing a class of check.

NORMALIZE_WHITESPACE collapses every run of whitespace, in both expected and actual, to a single space before comparing. It makes indentation and line wrapping irrelevant, which is what a reader needs when a value is a long nested structure. It also means the example no longer demonstrates line breaks, so an example about formatting cannot use the flag.

ELLIPSIS makes a literal ... in the expected output match any substring, including the empty string and including newlines. This is the only practical way to write an example containing a path, an object address, a hash, or a timestamp. The cost is that ... swallows everything between the surrounding fragments, so an example that elides the middle of a long message stops checking whether that middle is present, correct, or even sensible.

IGNORE_EXCEPTION_DETAIL relaxes the matching of an exception line, ignoring the module qualification of the exception class and the message text. Without it, an expected traceback must match the exception type and message exactly, and the traceback body itself is not compared at all: only the final line, of the form TypeError: unsupported operand type(s), is checked by default. That asymmetry is the single most surprising property of the format. A four-frame stack trace in the docstring looks thorough and verifies one line.

DONT_ACCEPT_TRUE_FOR_1 and DONT_ACCEPT_BLANKLINE disable two default leniencies: that 1 matches True, and that a blank line in the actual output matches <BLANKLINE>. Turning them on makes examples stricter, and the fact that both leniencies exist shows the pressure the format is under from Python 2-era transcripts and from print statements that produce blank lines.

Options are set globally through the runner — optionflags=doctest.ELLIPSIS in testmod, or -o ELLIPSIS on the python -m doctest command line — or per example through a directive appended to that example’s source line. The per-example form is the useful one, because a global ELLIPSIS silently weakens every example in the module.

Skipped examples are the drift the system cannot see

# doctest: +SKIP removes an example from execution. The reason in practice is almost always nondeterminism: a network call, a file that does not exist on the build machine, an example that depends on a random seed, a plot that opens a window. Skipping is honest about the limitation and dishonest about the consequence, because the skipped block is the only part of the file that can diverge freely while the rest of the suite stays green.

There is a middle category that the format cannot express: examples whose output is nondeterministic but whose behaviour is checkable. A function returning the current time produces an untestable transcript and a perfectly testable return type. The pattern that recovers these is to separate the illustration from the assertion: keep the transcript for the parts that are stable and move the rest into a test function, where the assertion can compare an interval rather than a string. The example then no longer claims to be executable, which is a loss of coverage and a gain in honesty, and contract-level descriptions can state the guarantee in prose where the string comparison cannot.

Collecting examples changes what a test suite can see

doctest.testmod() collects examples from every docstring in a module and runs them in that module’s globals. This last detail matters more than it appears. Examples share a namespace, so an example can use names defined by an earlier example in the same docstring, which makes a sequence of examples read as one session. It also means a failure in the first example usually causes the following ones to fail for want of their inputs, and the report shows a cascade whose root cause is one line earlier.

Under unittest, doctest.DocTestSuite wraps each docstring’s examples as a test case, so a failure is reported with the usual machinery and can be selected by name. The unit of granularity is the docstring, not the example, which is coarse: a forty-line tutorial docstring with one unverifiable line cannot be partially run.

Sphinx’s doctest builder changes the unit again by collecting examples from the documentation sources rather than from Python docstrings, including doctest blocks in reStructuredText files and code in .. testcode:: directives. That path is worth knowing because it is the only way a hand-written page, rather than a docstring, participates. The pages that use it tend to be the ones where the example is the point: a quickstart, a configuration walkthrough, a migration guide.

What the format is actually buying

The alternative to an executed example is an unexecuted one, and the failure mode of the unexecuted kind is slow and quiet. A signature change renames a parameter; the prose still says the old name; the example still parses and still looks right; nothing fails until a reader copies it. Executed examples convert that class of error into a test failure at the moment the change lands, in the same commit as the diff that caused it.

The cost is real and should be stated plainly. Every example becomes a compatibility constraint, so a deliberate behaviour change requires editing every transcript that shows the old behaviour, and a large documentation set with many examples can make a small output change look enormous. Examples must also run somewhere: a build that executes documentation needs the library importable and any example dependencies installed, which is a weaker environment than a unit-test runner usually has, and examples that touch the filesystem write to the working directory of the build.

Those costs are bounded and visible. The unbounded cost is the skipped and non-executable fraction, which looks like coverage in a rendered page and behaves like prose. Measuring that fraction, rather than counting examples, is the honest summary of how much of the documentation is verified, and it is the same question that deterministic build checks answer for every other kind of claim in a page.