What an Automated Accessibility Check Can Decide

An audit tool reports a list of violations and a list of elements it could not decide. The second list is usually longer than the first, and experienced testers spend more time on it. The split is not a limitation of a particular product. It follows from what a rule can see and from the shape of the success criteria themselves.

A rule engine works on a serialized tree, a set of attributes, and computed styles. Every question it answers is a predicate over that data: does this element have a role, is its accessible name empty, is this ratio below 4.5:1, is this attribute allowed on this role. Questions that need the meaning of the content, or the effect on a person completing a task, cannot be expressed that way. The two classes are not cleanly separated in the guidelines, which is why tools disagree about coverage and why a passing run is weaker evidence than it appears.

Predicates over the serialized tree

The decidable class is larger than it first appears, because a great deal of the specification is mechanical. An image either has an alt attribute or it does not. A form control either has an associated label or it does not, where “associated” means the label’s for matches the control’s id or the control sits inside the label element. A page either declares a language on the root element or it does not. An id either occurs once in the document or several times. An ARIA role either exists in the vocabulary or it does not, and each role either permits an attribute or places it in a prohibited set.

Each of these is a statement about the document as parsed, and each maps to a success criterion closely enough that a rule can be written for it. Language of page, 3.1.1, is a single attribute check. Name, role, value, 4.1.2, becomes a question about whether the computed role supports a name and whether the name computation returns an empty string. Info and relationships, 1.3.1, is only partly decidable, but the parts that are decidable, such as a table header cell marked with <th> rather than styled bold, a list of links marked up as a list, or a heading marked with a heading element, are exactly the parts developers most often get wrong.

The measured criteria form a second decidable group. Contrast is arithmetic on colour values. Target size is arithmetic on bounding boxes. Text spacing is arithmetic on computed line height, letter spacing, and word spacing. Reflow is arithmetic on scroll width against viewport width. Once a quantity is defined, a threshold can be applied mechanically, and the interesting engineering problem moves to measurement: which pixels belong to the text, which element paints the background, what happens when the background is an image.

A third group is decidable only in the negative direction. A rule can prove that an input has no accessible name; it cannot prove that the name the author wrote is meaningful. It can prove that a div with an onclick handler is not keyboard reachable; it cannot prove that the handler’s effect makes sense to a keyboard user. Asymmetric evidence like this is the normal case, not the exception.

The judging layer

Some criteria are written in terms of purpose, and purpose lives in the content. Link purpose in context asks whether the link text, together with its programmatic context, describes the destination. A rule can detect the degenerate cases, such as text that is only “click here”, text that is empty, or text duplicated across several links to different destinations, but the check that matters is whether a person reading the list of links can predict where each one goes.

The same applies to alternative text. The mechanical check is that alt exists and is not obviously a filename; the judgement is whether the sentence describes the image’s function in that place. An image that illustrates a paragraph needs a description; the same image next to a caption that already says everything needs an empty alt so it is skipped. Both are correct markup, and no rule can distinguish them without reading the surrounding prose.

Order and sequence are similar. The reading order of the document is programmatic and inspectable, but whether that order conveys the intended meaning depends on the visual layout the author was trying to produce. A rule cannot see the design. It can flag likely mismatches, such as content positioned with floats or absolute positioning that no longer follows source order, but the verdict belongs to a person comparing the two renderings.

Error identification is another criterion that looks mechanical and is not. A rule can confirm that an error message is programmatically associated with the field. It cannot confirm that the message identifies the error and suggests a correction, which is what the criterion asks.

Why tools report “incomplete”

Engines encode the boundary explicitly. A rule whose outcome depends on data the engine cannot resolve returns neither a pass nor a violation but a third state, usually called incomplete or needs review. The common triggers are well known: text drawn over a background image or a gradient, where the effective background colour is not derivable from the cascade; an element hidden by an animation or an off-screen transform at the moment of measurement; a name that resolves from an aria-labelledby reference into a shadow root the engine cannot traverse.

The third state is not a defect in the tool. It is the honest answer, and it is the part of the report that requires a person. A run that reports twelve violations and forty incomplete results has identified twelve defects and raised forty questions.

The distinction between incomplete and inapplicable matters more than it looks. Inapplicable means the rule’s selector matched nothing: no images on the page, so the image-alt rule was not evaluated. A dashboard that shows “0 failures” for that rule is telling the truth about a rule that never ran. When the same dashboard aggregates across a site, pages without images dilute the picture, and a rule that only fires on a rare widget appears to pass everywhere.

// An empty alt and a missing alt are different assertions, not the same value.
const findings = [];
for (const img of document.querySelectorAll('img')) {
  if (!img.hasAttribute('alt')) {
    // Nothing was asserted; the image is exposed with its file name or URL.
    findings.push({ img, kind: 'missing-alt' });
  } else if (img.getAttribute('alt') === '') {
    // Explicitly decorative: correct only if the image carries no information.
    findings.push({ img, kind: 'decorative-alt-needs-review' });
  }
}

What a threshold hides

A measured criterion is a comparison against a fixed number, and the number is chosen so that the criterion is testable, not so that it is sufficient. Contrast at 4.5:1 is the threshold for body text in the minimum level; text at 3:1 is permitted when it is large, defined as at least 18 point, or 14 point when bold. Passing at 4.6:1 is conforming and can still be hard to read for someone with low contrast sensitivity, particularly on a dim display or through a reflection.

The same structure appears elsewhere. The reflow criterion sets 320 CSS pixels, the equivalent of a 1280-pixel viewport at 400 percent zoom. A layout that fits at 321 pixels passes and is not thereby usable. The target size criterion at the minimum level sets a 24 by 24 CSS pixel box, with exceptions for inline links in a sentence and for targets whose spacing provides an equivalent. All of these are floors, and the test reports the floor, not the experience.

Testers who treat the threshold as the goal end up optimizing the measurement. The common form is nudging a grey from just below 4.5:1 to just above it while leaving the type at a weight and size that makes the ratio irrelevant.

Splitting the work between engine and person

The useful division follows the two classes. The engine owns everything that is a predicate: attribute presence, role validity, name emptiness, ratio arithmetic, duplicate identifiers, missing language, focusability of handlers, heading and landmark structure, and the full matrix of ARIA attribute permission. These are cheap to check, exhaustive across a large site, and boring to check by hand, which is precisely why they should be automated. The coverage question is how much of the guideline set that class actually spans, and it is smaller than a clean report suggests.

A person owns everything that is a judgement: whether a name describes the thing, whether an order tells the right story, whether an error message helps, whether a custom widget’s states make sense to someone who cannot see it. These are expensive, so they are sampled rather than exhaustive, and the sampling has to be justified. The accessibility tree is the shared vocabulary between the two sides: it is what the engine reads and what the person inspects, which is why tests on either side should be phrased in terms of it.