Why Zero Violations Is Not a Pass

A page returns no violations from a rule engine. The same page, opened with a keyboard, traps focus inside a date picker. Both statements are true and neither contradicts the other. The engine never pressed a key.

The gap between a clean automated report and actual conformance has three distinct components, and they are worth separating because each is closed by a different kind of work: the criteria the engine cannot express at all, the criteria it can express but could not resolve on this page, and the criteria that only exist once a user interacts with the document.

The size of the expressible subset

Success criteria are written for human evaluators. Many are phrased in terms of what a person can perceive or understand, and no predicate over the DOM captures them. The set that maps cleanly to rules is a minority of the total: attribute presence, role validity, name emptiness, arithmetic thresholds, structural relations.

Several entire categories are close to unreachable. Adaptable criteria beyond the mechanical parts require knowing whether the structure chosen matches the meaning of the content. A grid of div elements with correct grid roles conveys a data table, and a real <table> used to position two blocks of text conveys nothing; deciding which of the two is which depends on what the author meant the content to be. Understandable criteria are about wording: whether labels and instructions are clear, whether the reading level is appropriate, whether navigation is consistent in a way that helps rather than confuses. Robust criteria are partly mechanical, but the interesting failures, such as a custom widget whose state is announced wrongly, need the widget exercised.

Even inside the mechanical subset, coverage is uneven across the four principles. A rough picture: the largest share of automatable checks sits in perceivable and robust, a moderate share in operable, and the smallest in understandable. Any statement of the form “automation catches X percent of issues” depends on how issues are counted: per criterion, per failure instance, or per page. That is why published percentages differ so widely and why none of them is a useful target.

The honest formulation is directional. Automated checks are very good at catching a specific family of defects: unnamed controls, missing alternatives, invalid ARIA, insufficient contrast in flat backgrounds, duplicate IDs, missing language, keyboard handlers on non-interactive elements, skipped heading levels where the rule is enabled. They are weak at everything that requires reading the page as a person would.

The incomplete queue as a work item

The second component is the set of elements the engine examined and could not decide. On a page with layered backgrounds, custom controls, and dynamic regions, this list can be longer than the violation list, and it is frequently ignored because it does not fail the build.

The contents are informative. A contrast check that returns incomplete because the text sits on a photographic hero image is reporting a place where contrast is genuinely unknown until someone looks. A name check that returns incomplete because the name comes from an aria-labelledby pointing at an element the engine could not traverse is reporting a reference graph more complex than the tool’s model.

Working the queue means resolving each entry by hand and recording the resolution. Two outcomes are possible: the element is fine, in which case the resolution is a note attached to that selector so that the next run does not re-raise it, or the element is not fine, in which case it becomes a defect. Both outcomes are work; only the second produces a violation.

An exception list is a maintenance liability unless it is keyed to something stable. Keying to a CSS selector path breaks on any structural refactor. Keying to a rule plus a human-written justification plus a review date survives refactors and forces periodic re-examination, which matters because the reason for an exception often expires: the hero image is replaced, the gradient is removed, the third-party widget is upgraded.

Interaction as a coverage boundary

The third component is the largest and the least visible in a report. A static analysis of the initial document sees one state of the page. A page with a menu, a dialog, a tab panel, an accordion, a form with validation, and a live region has many states, and most defects live in the transitions.

Consider what a keyboard user does that the initial DOM does not reveal: tab through the document, and focus may land on an element that is invisible, or skip an element that is visible, or enter a region and never leave it. Activate a control with Enter and the resulting content may be inserted with no announcement. Open a dialog and focus may stay behind it. Submit a form with errors and the first invalid field may not receive focus, so the error summary is announced while the user has no way to reach the offending field except by tabbing from the top.

None of these are detectable without driving the page. The tests that find them are sequences, and sequences are what a focus-order protocol is built to record. A rule engine evaluates the document; it does not have a model of the user’s position in it, which is exactly the state that keyboard interaction manipulates.

What a clean run does establish

The value of a zero-violation result is a bound, not a proof. It establishes that a specific list of mechanical defects is absent from the states that were scanned. That is a real and useful guarantee, and it is cheap to obtain on every commit, which is why it belongs in continuous integration rather than in a quarterly audit.

It also establishes a floor on the cost of the remaining work. If the run scanned 400 pages and reported nothing, then 400 pages are free of that list of defects, and a manual sample can concentrate on the criteria the engine did not attempt. The sample still has to be drawn with the conformance methodology in mind, because a sample drawn from the most-visited pages will miss the template that a single content editor uses.

Three properties of the run determine how much the bound is worth, and all three are usually unstated in a report: the rule set and its version, whether the scan covered rendered states or only the initial document, and what happened to the incomplete results. A report that omits these is a number without a meaning.

Composing the evidence

The practical arrangement is a three-layer test suite with different cadences and different failure semantics.

Layer one is the static scan, run on every build, failing on violations in the criterion-mapped rule set. It is fast, deterministic, and its failures are almost always real. Its configuration is pinned and its suppressions are reviewed.

Layer two is the scripted interaction suite, run on the same cadence where it is affordable: keyboard traversal of the main flows, dialog focus management, form error handling, and the announcement of dynamically inserted content. These are ordinary browser tests with accessibility assertions, and their oracles are the accessibility tree and, for some, the speech output a screen reader produces.

Layer three is human evaluation of a sample, chosen by template and by function rather than by traffic, covering the criteria that no script can decide: whether names describe, whether order tells the right story, whether instructions are understandable, whether a widget makes sense without sight. It is slow, so it is sampled, and the sample is documented so that the claim it supports is bounded.

The three layers answer different questions and their results are not additive. A page can pass layer one and fail layer two; a page can pass both and fail layer three. The claim that can be made at the end is a claim about which layers were run, over which pages, at which version. That is the form a conformance statement takes, and the reason a single “accessibility score” is not one.