Deterministic Checks That Keep a Documentation Build Honest

A documentation build that only renders pages will succeed on a site where every internal link is broken, every example is stale, and the product is spelled three ways. Rendering proves that the markup parsed. Everything else about the document is a separate assertion, and the value of adding those assertions depends less on how many are added than on whether each one is decidable.

Decidability is the property that separates a check from an opinion. “This paragraph is unclear” cannot be answered by a program and should not be a build gate. “This link target does not exist in the built output” can be answered exactly, on every run, with no exceptions. The useful discipline is to admit only the second kind into the gate, and to route the first kind to review where a human can weigh context.

Four families of assertion, ordered by strength

The checks that survive in practice fall into four groups, and their strength decreases in a specific order.

Structural assertions are the strongest. Frontmatter parses as YAML; required keys are present; domain and cluster match the directory; slugs are unique; every page is reachable from the navigation; heading levels do not skip from ## to ####. Each of these has a single correct answer, and each failure has a mechanical fix. They are also the cheapest to implement, since they operate on the source tree rather than on rendered output.

import { readFileSync, readdirSync } from 'node:fs';

const slugs = new Set();
for (const file of readdirSync('docs')) {
  if (!file.endsWith('.md')) continue;
  const text = readFileSync(`docs/${file}`, 'utf8');
  const front = /^---\n([\s\S]*?)\n---/.exec(text);
  if (!front) throw new Error(`no frontmatter: ${file}`);
  const slug = file.slice(0, -3);
  if (slugs.has(slug)) throw new Error(`duplicate slug: ${slug}`);
  slugs.add(slug);
  if (![...text.matchAll(/^## /gm)].length) throw new Error(`no sections: ${file}`);
}

The snippet is deliberately dull. That is the property that makes it run on every change instead of on a schedule.

Referential assertions come next: every internal link resolves to an existing output path, every image file exists, every include directive names a real file, every cross-reference label is defined. These require the build output rather than the source, because paths in the output are a function of the site’s URL scheme, and because generated ids do not exist in the source at all. Fragment links belong here, and they are the ones most often omitted by a checker that only requests URLs.

Executable assertions are next: examples run and produce the output they claim. Their strength is high and their scope is narrow, because an example covers one call path with one set of inputs, and the transcript format only verifies the parts of behaviour that render as text.

Lexical assertions are the weakest and the most numerous. They match patterns in prose: a banned term, an inconsistent spelling of a product name, a doubled word, an undefined abbreviation. They are useful precisely because they are crude; a rule that tries to judge sentence quality produces false positives that teach authors to ignore the tool.

External links have three outcomes, not two

Internal link checking is deterministic: the output is on disk, and a missing path is missing. External checking is not, and treating it as though it were is the main reason link gates get disabled.

An external request produces a success, a confirmed failure, or an unknown. A 404 or 410 is a confirmed failure: the server answered, and its answer is that no representation exists at that URL, with 410 stating that the condition is permanent. A timeout, a connection reset, a 403 produced by bot filtering, and a 429 from a rate limiter are all unknown: the server may serve the same URL to a browser without difficulty. A check that reports unknowns as failures will fail on servers that block automated clients, which is a large and growing set.

The arrangement that keeps the signal is to run internal checks in the build gate and external checks on a schedule, with a retry and a small delay between requests, and to treat only confirmed failures as reportable. The schedule matters because external rot is not caused by the change under review; a page that was fine last week can be broken by someone else’s site rewrite, and blocking an unrelated pull request for that is a policy that will be circumvented.

External checks also need a suppression mechanism with an expiry, because a permanently suppressed link is an unverified link. A suppression entry that carries the date it was added and the reason is auditable; a global allowlist of domains is not.

Prose rules earn their place by having an unambiguous trigger

A prose linter is a set of patterns applied to text, and its output is only as good as the specificity of those patterns. The rules that work share a property: a competent author, shown the match, agrees that it is wrong without needing the surrounding argument.

Terminology consistency is the highest-value class. If the product is written PostgreSQL in one page and Postgres in another, both are defensible English and the inconsistency is a real defect for the reader who searches for one and lands on neither. A rule that fixes the canonical spelling in a terminology file, and flags variants, is mechanical and has almost no false positives. The same file feeds the search index’s synonym list, which is why terminology discipline and search relevance are the same problem seen from two ends.

Spelling checks need a domain dictionary and a rule that skips code spans. Without the skip, every identifier becomes either a spelling error or a dictionary entry, and the dictionary grows until it accepts everything. With the skip, the dictionary stays small enough to review.

Rules that fire on sentence structure — passive voice, nominalisations, sentence length — have a much worse ratio. They fire on correct sentences, authors learn to rephrase mechanically to silence them, and the resulting prose is not better. If such a rule is wanted at all, it belongs in an advisory report with a visible count, not in the gate.

Every rule needs an escape hatch that is local and explicit: a comment marker on the line, with a required reason. A per-file disable is the failure mode to avoid, because one disable at the top of a file removes the rule from everything below it, including lines added later by someone who never saw the decision.

The zero-warning baseline

A build that emits warnings and exits zero trains its readers to ignore the output. The count grows, the newest warning is invisible among the old ones, and the assertion has effectively been removed while appearing to exist.

Two policies repair this. The first is warnings-as-errors, available in most documentation toolchains as a flag, which is only viable once the warning count is zero. The second is a ratchet: record the current count, fail when it increases, and require the count to fall when a file is touched. The ratchet is the pragmatic choice for a large existing corpus, because it makes progress possible without a cleanup project that blocks all other work.

The ratchet has a cost of its own: it needs a stored baseline, and the baseline must be regenerated deliberately rather than automatically, or it becomes a record of whatever the build last emitted. It also interacts badly with non-deterministic output, where the count varies between runs. Which leads to the property underneath all of this.

The build itself has to be deterministic

A check is only meaningful if the same input produces the same result. Three sources of variation break that.

Iteration order: a generator that walks a set or an unordered mapping produces member ordering that changes between runs, which makes generated pages differ without any source change and makes diffs unreviewable. Sorting by a stable key at the point of emission fixes it.

Embedded time and environment: a footer containing the build timestamp, an absolute source path in an error message, or a version string read from the environment all make two builds of identical sources differ. A reproducible build does not need to be bit-identical for its own sake; it needs to be identical so that a difference means a change.

Network access during the build: fetching an external inventory, a remote schema, or a font makes the build’s success depend on a third party. Vendoring the artifact, or making the fetch a separate step with a recorded checksum, restores determinism.

Once the build is deterministic, a further check becomes available: the rendered output can be compared against the previous release, and any change to a published page that was not accompanied by a change to its source is a signal worth investigating. That comparison is the mechanical half of drift detection, which is otherwise a question about meaning rather than bytes.

What should stay out of the gate

Two classes of check are attractive and should not block a merge.

Freshness heuristics are the first. “This page has not been edited in two years” is a fact about the repository, not about the page; a stable specification can be correct and untouched, and a page edited yesterday can be wrong. The heuristic is useful as a report that orders a review queue, where a human can look at the page and decide, and harmful as a gate that forces a cosmetic edit to reset a timer.

Length and coverage targets are the second. A word-count minimum produces padding, and a documentation-coverage percentage measured by docstring presence produces one-line docstrings. The measurable version of the same concern is narrower and better: which public symbols have no entry at all, which entries lack a description, and which examples are skipped.

The test for any proposed check is whether its failure names an action. A broken link names the file and the target. A missing reference names the label. A terminology violation names the canonical form. A warning that says a page is old names nothing that a reader can act on, and it belongs in a report rather than in the path of a change.