How an Accessibility Rule Engine Decides

Two checkers run against the same page and produce different counts. One reports a colour-contrast failure that the other does not see; the other flags a heading-order jump that the first ignores. Neither is malfunctioning. They differ in which success criteria they attempted, how they modelled the cascade, and what they chose to do with a result they could not resolve. Understanding the stages a rule passes through makes those differences predictable instead of mysterious.

The architecture is consistent across the mature engines, and axe-core is the one whose internals are documented and whose rule set is inspectable, so it serves as the reference implementation here.

Rules, checks, and the two-level structure

A rule in axe-core is a named bundle: an identifier, a selector that limits the candidate elements, a mapping to one or more success criteria, a set of tags, and a set of checks grouped by the outcome they argue for. A check is the unit that actually evaluates: it takes an element, optionally its subtree, and returns a boolean or the string undefined for “cannot determine”.

The split exists because a single criterion is rarely decided by one test. A rule for “buttons must have a name” might have one check that looks for aria-label, another that looks at the computed name, and a third that inspects the subtree for text. Checks are grouped into all (every one must pass), any (at least one must pass), and none (none may pass). The rule’s verdict is a fold over those three groups: all checks are evaluated first, then any, then none, and the first group that settles the outcome short-circuits the rest.

This structure is what produces the third state. If an any group contains checks that return undefined and none that return true, the rule cannot be failed and cannot be passed, so it reports incomplete. A none group with an undefined check behaves the same way. The engine is explicit about this: undefined propagates upward as uncertainty rather than being coerced to false.

The evaluation pipeline

The engine’s run has four phases, and each one is where a specific class of disagreement between tools originates.

Parsing and preprocessing come first. The engine reads the DOM as the browser parsed it, applies its own normalizations, and computes what it needs: resolved URLs for href, the computed accessible name for candidate elements, and for contrast, the effective background colour behind a run of text. That last one is the expensive part. It requires walking ancestors, accumulating background colours with alpha compositing, and stopping when an opaque colour is found; when an ancestor paints an image, the value is not derivable and the check returns undefined.

Matching is second. Every rule carries a selector, and the engine evaluates all of them against the document, typically with querySelectorAll-style matching extended to shadow roots where the engine supports them. An element can be a candidate for many rules. The set of candidates for a rule is not the same as the set of elements that violate it: a rule that selects all img elements will be inapplicable on a page whose images all have alt.

Evaluation is third, and it is where checks run in document order, with results cached per element and per check. Caching matters for performance and it has a semantic consequence: a check that depends on state the page mutates during the run sees the state as of first evaluation. Engines that support running against a live page therefore either freeze the page or accept the inconsistency.

Reporting is last. Results are assembled per rule and per element, deduplicated, and given a node target: a CSS selector, an XPath, and a snippet of the element’s HTML. The node target is the bridge to a human: it is what a report links to and what an exception list keys on.

// A minimal check with the three-valued contract the engine relies on.
function checkImageAlt(node) {
  if (!node.hasAttribute('alt')) return false;      // violation
  if (node.getAttribute('alt').trim() === '') return true; // decorative, permitted
  return undefined;                                  // present but unjudged
}

The undefined in that example is not a cop-out. An image with a non-empty alt may still have an alt that duplicates the caption, names the file, or describes the wrong thing. Returning true would let a rule claim a pass it cannot support; returning false would report a violation that may not exist. The third value is the only honest one, and it is what puts the element into the review queue.

Impact levels and why counts mislead

Each rule declares an impact: minor, moderate, serious, or critical. The declaration is a judgement by the rule author about the consequence of the violation, not a measurement. A missing document language is typically minor; a control with no accessible name is typically critical. The impact is attached to the rule, so every violation of that rule carries the same level regardless of context, which is why a critical violation in a footer is ranked alongside a critical violation in the main task flow.

The tag recording a rule’s maturity is the more useful signal and the less commonly read one. Rules carry tags, among them one marking a rule as experimental, and rules carrying that tag are excluded from the default run. A tool that runs the full set including experimental rules will report more findings, and some of those findings will be withdrawn in a later release when the rule is refined. Comparing two tools by total count therefore compares two different rule sets, and comparing two versions of one tool compares two different snapshots of a moving target.

Best-practice rules are the third axis. These do not map to a success criterion; they encode guidance that the authors believe improves usability, such as a page having a single main landmark or headings not skipping levels. A report that mixes them with criterion-mapped rules makes a page look worse than its conformance status warrants. The filter that matters for a conformance claim is the one that restricts to WCAG-mapped rules at a chosen level.

Where engines disagree about the same element

Contrast is the classic case. The ratio itself is arithmetic defined by the relative luminance formula in the specification: linearize each sRGB channel, weight the channels, and compare the two luminances as a ratio. The arithmetic is not where tools diverge. The divergence is in identifying the foreground and background colours. Text rendered with a gradient, an image behind it, a background whose alpha is composited over another background, or a text colour that is inherited through several layers of custom properties can all be resolved differently by different implementations. One engine may compute a ratio from the nearest opaque ancestor; another may give up and report incomplete.

Name computation is the second case. The accessible name algorithm is specified, but its inputs include CSS-generated content, aria-labelledby traversal across shadow boundaries, and the treatment of whitespace between concatenated fragments. An engine that reads the DOM attribute instead of performing the full computation will disagree with one that performs it, and both will disagree with the browser’s own tree in edge cases. The test oracle question is precisely whether the engine’s answer matches the browser’s.

Third is the treatment of hidden content. A rule engine has to decide which elements count: display: none is straightforward, visibility: hidden is not, off-screen positioning is a judgement, and content inside a collapsed <details> element is hidden from the user but present in the DOM. Each engine encodes a policy, and a page whose only problems are inside collapsed regions will get different reports from two engines that agree on everything visible.

Treating the engine as a component

The practical consequence is that an engine’s output is a function of its version, its rule configuration, and its treatment of uncertainty. A test suite that consumes it should pin all three. Pinning the version is ordinary dependency hygiene. Pinning the configuration means making the rule set explicit, covering which tags, which WCAG levels, and whether best-practice rules are included, because the default set changes between releases. Pinning the treatment of uncertainty means deciding whether incomplete results fail a build, and if they do, how the queue is worked down rather than muted.

A report that is treated as a stable contract is usable as a regression gate. A report that is treated as a score is not, because the score moves when the rule set moves and it says nothing about the parts of the guideline that the engine did not attempt. The size of that remainder is the subject of the coverage problem.