Detecting Drift Between Reference Material and the Code It Describes

Drift is usually described as a property of a document: a page that used to be right and is no longer. That framing hides the useful structure, because a document does not become stale as a whole. It contains dozens of individual claims, and they decay at different rates. A page can have a perfectly current signature, a parameter table that lost one entry, an example that still runs, and a paragraph explaining a retry policy that was replaced two releases ago. Any detector worth building has to operate on claims, because the file as a unit either passes or fails for reasons unrelated to any of them.

The direction of drift is also not uniform. Code changes and documentation stands still, but documentation can change while code does not, usually by being edited to describe an intention that was never implemented. The second case is harder to detect because the document is newer than the code it contradicts, which defeats every timestamp-based heuristic.

Two descriptions of the same interface

A reference page and an implementation are both descriptions of one interface, produced by different processes. The implementation is authoritative and executable; the page is derived and rendered. Drift is a disagreement between the two, and the detector’s job is to find disagreements that a machine can evaluate.

That framing immediately sorts the problem. Some disagreements are decidable by comparing two artifacts: a parameter appears in the signature and not in the table, a link refers to a section that no longer exists, an example produces different output. Others are decidable only against the world: whether the described retry policy is the one implemented, whether the stated complexity bound holds, whether the thread-safety claim is true. The first category is cheap to check continuously. The second is a review question, and the only way to make it cheaper is to put the contract elements where a reviewer will see them next to the diff.

Only some claims are machine-checkable

The checkable subset is larger than it looks, and it is worth enumerating because each item corresponds to a class of bug that otherwise ships.

Names and signatures are checkable when the page is generated from source or when the docstring declares parameters in a parseable form. Parameter names, their order, their defaults, and their presence are all comparable.

Examples are checkable when they are executed, which is the entire premise of documentation tests. An example that runs and produces the stated output is verified for the inputs it uses and nothing else.

Links are checkable, including fragments, against the built output. A reference to a symbol that was removed resolves to nothing, which is a direct signal that a page describes an interface that no longer exists.

Vocabulary is checkable when there is a registry to check against. Every flag, environment variable, configuration key, error code, and endpoint mentioned in prose can be extracted as a token and compared with the set the code defines. This is the highest-yield mechanical check for operational documentation, because those tokens are exactly the ones a reader copies.

Structural claims about the interface are checkable in narrow cases: if the page says a function raises two exception types and the docstring declares three, a comparison can flag the difference. It cannot tell which side is right.

What remains is the majority of the prose: rationale, ordering constraints that span several calls, guarantees about concurrency, migration advice, and the explanation of why an alternative design was rejected. None of that is derivable from the code, and none of it can be diffed against it.

Timestamp heuristics and their two failure directions

The cheapest drift detector is a comparison of modification times: if the code changed after the documentation, flag the documentation. Repositories make this easy, since every file has a last-commit date, and the check requires no parsing.

It fails in both directions, and the failures are structural rather than tunable.

False positives come from changes that do not affect the described behaviour. A reformat, a comment edit, a dependency bump in the same directory, or a rename of a local variable all update the code’s timestamp and none of them change the interface. A team that receives a steady stream of these flags learns to ignore the report, and the report is then worthless for the cases that matter.

False negatives come from the mapping between files. Documentation for a module is often not stored beside it; a page about the HTTP client may be maintained in a separate directory, and the code that determines its behaviour may live in three files, none of which is named in the page. A timestamp comparison over unrelated paths cannot see the relationship at all. Deliberate changes make it worse: an author can fix a typo in a page and reset its timestamp without touching a single claim.

The refinement that helps is an explicit mapping from a page to the code it describes, declared in the page’s metadata and used to select what counts as a relevant change. It converts an unreliable global heuristic into a reliable local one, at the cost of maintaining the mapping. It also makes ownership visible: a change to a mapped path can request review from the page’s owner, which is often more valuable than the flag itself.

Comparing declared parameters against the real signature

Where documentation declares parameters in a structured form, the comparison is a few lines, and it catches the most common form of reference rot: a parameter added to the code and never added to the table.

import inspect, re

def documented(doc):
    """Names declared in a Sphinx-style field list."""
    return set(re.findall(r'^:param ([A-Za-z_]\w*):', doc or '', re.M))

def compare(fn):
    real = set(inspect.signature(fn).parameters)
    stated = documented(inspect.getdoc(fn))
    return {'undocumented': real - stated, 'stale': stated - real}

def open_connection(host, port=5432):
    """Open a connection.

    :param host: server name
    """
    return host, port

print(compare(open_connection))

The stale set is the interesting one. A parameter documented but absent from the signature is either a removal that the page missed or a documentation error that has been there since the page was written; both are worth failing a build over, and both are invisible to a reader who trusts the table.

The same comparison generalises to defaults and to type annotations, with a caveat. A default rendered as an expression can differ textually while being semantically identical, and a documented type written as prose cannot be compared at all. Comparisons should be limited to cases where both sides are structured, and the check should report a difference rather than attempt to judge it.

For C and C++ the equivalent comparison runs against parsed headers, and it has to account for the fact that the declaration and the definition may carry different amounts of information. For generated reference pages the comparison is unnecessary, since the page is produced from the same declaration; the drift moves into the prose, where the generator cannot help.

Checking vocabulary against the code’s own registry

Flags, keys, and codes are finite, enumerable, and written in a distinctive form, which makes them the best candidates for a mechanical check.

The mechanism has three steps. Extract candidate tokens from the rendered page using patterns that match the interface’s conventions: a leading -- for command-line flags, an uppercase identifier for error codes, a dotted or dashed key for configuration. Collect the authoritative set from the implementation: the argument parser’s registered options, the configuration schema’s properties, the error enum. Compare, and report tokens that appear in prose but not in the code.

The check catches a specific and painful class of error: documentation for a flag that was renamed or removed. It does not catch a flag that exists and is described incorrectly, and it does not catch a missing description of a flag that exists, unless the comparison is run in both directions. The reverse direction is worth adding, with an allowlist for internal options that are deliberately undocumented.

Two implementation details decide whether the check survives contact with real pages. The extraction must ignore code blocks that show output or unrelated examples, or it will report tokens from a shell transcript as documentation of this interface. And the comparison must be scoped per page or per product, since a page about one service legitimately mentions another service’s flags.

What passes every check and is still wrong

The residual after all mechanical checks pass is large, and it is worth being precise about its shape.

A page can name every current parameter, run every example, resolve every link, and still be wrong about the order in which two calls must be made, about what happens when a retry is interrupted, or about the memory cost of a large input. These are the claims that make the documentation useful, and they have no machine representation to compare against.

The only mechanism that addresses them is proximity: the person changing the behaviour sees the page that describes it while the change is in hand. That means documentation diffs reviewed in the same change as the code, a convention that a page must be either updated or explicitly declared unaffected, and a mapping from code paths to pages so that the reviewer can find the second without searching. Where a change is large enough to invalidate a decision, the record of the decision belongs in an architecture note rather than in the reference page, and the reference page should link to it rather than restate it.

None of this makes drift disappear. It moves the detectable part into a build that fails, which converts a slow, reader-discovered failure into a fast, author-visible one, and it leaves the undetectable part to a reviewer who at least has the two descriptions side by side.