Conformance Evidence and Defect Triage

The output of all the preceding techniques is a pile of results: scan violations, incomplete entries, focus sequences, layout stress failures, widget contract results, and manual findings. A conformance claim is a statement about that pile, and the statement is only as good as its scope, its sampling, and its record of what was excluded.

The methodology that exists for this is WCAG-EM, and its structure is worth following because it forces the three decisions that informal audits skip: what counts as the evaluated set, how the sample is drawn from it, and what the claim covers.

Defining the evaluated set

A site is not a document. It is a set of URLs produced by a small number of templates over a large space of content, plus a set of states produced by interaction. The evaluated set has to be defined before sampling, and the definition is where most claims become unfalsifiable.

Three levels of definition are useful. The first is the origin: every URL under the domain, including the ones not linked from anywhere. The second is the template: a distinct page structure with its own markup, such as a product page, a search results page, a checkout step, an article page, an error page. The third is the state: a template with a menu open, with a validation error, with an empty result set.

The first level is what a claim usually names and what a sample cannot actually cover. The second is what a sample is drawn from, because defects cluster in templates: a missing label on a search field is one defect repeated across every page that includes the header, and a broken table header is one defect repeated across every instance of one template. The third is where the largest share of real defects lives and where the sample is thinnest.

A defensible claim states which of the three it covers. “Every template in the evaluated set, in its default and error states, was evaluated” is a claim that can be checked. “The site is accessible” is not.

Drawing the sample

WCAG-EM’s sampling procedure is structured: include pages that are structurally different, pages that represent common content, pages that are essential to the process being evaluated, and pages that are randomly selected from the remainder. The random component is what makes the sample’s coverage claim meaningful, and it is the component most often dropped.

The template-based version of the same idea is more practical for a large site and equally defensible if it is documented. Enumerate the templates by clustering URLs by their rendered structure, or by reading the routing table if the site’s generation is known. Take at least one instance of every template, and for the templates with the most instances or the most interaction, take several: the default state, an empty state, a full state, and an error state.

Two properties of the sample need recording. The first is the ratio: how many instances were evaluated against how many exist, per template. A template with one instance evaluated out of one is fully covered; a template with one instance out of forty thousand is a sample, and the claim should say so. The second is the selection rule: which instances and why. “The first instance returned by the search endpoint” is a rule, and it is biased toward whatever the endpoint ranks first, which is worth knowing.

The sample’s size is a judgement. The methodology does not fix a number, and the useful discipline is that the sample grows with the diversity of the templates rather than with the size of the site.

Recording scope, level, and exceptions

A conformance claim has four parameters, and omitting any of them makes it uninterpretable.

The guidelines version comes first, since criteria are added and removed between versions, and a page can conform to one version and not another. The conformance level comes second: A, AA, or AAA. AA is the common target, and a claim of AA is a claim about a specific set of criteria, not about accessibility in general. AAA is rarely claimed in full because several criteria at that level cannot be satisfied for all content.

The scope comes third: which pages, which states, which parts of the page. A common and legitimate scope limit is third-party content that the site does not control, which the guidelines address with a statement of partial conformance. A claim that excludes an embedded payment iframe must say so.

The fourth parameter is the exception list. Every resolved incomplete entry, every known defect that is accepted rather than fixed, and every place where the criterion was judged satisfied by a documented interpretation belongs here, with a justification and a review date. The list is the difference between a claim and a marketing statement, and it is the artifact that survives staff turnover.

Ranking by user impact

A defect list ranked by rule severity is ranked by the rule author’s guess about typical consequence. A defect list ranked by user impact is ranked by what the defect prevents on this site, for which users, on which path.

The useful axes are reach and severity. Reach is the number of pages or flows the defect affects, which is derivable from the template analysis: a header defect affects every page, a checkout-step defect affects one path, and a defect in a rarely used template affects few. Severity is the degree of blockage: a control that cannot be reached by keyboard blocks a task for keyboard users; a control that is reachable but announces nothing is usable by some and not by others; a control whose name is terse but accurate is a nuisance.

Two combinations dominate the queue. High reach with high severity, such as a keyboard trap in the global navigation, is fixed first because it blocks a task on every page. Low reach with high severity, such as a form that cannot be completed without a mouse, is fixed next because it blocks a task completely, even though few encounter it. High reach with low severity is a large mechanical change and often the cheapest to make, which is why it tends to be batched.

The axis that is usually missing is who is affected. A defect in the tab order affects keyboard users, which includes many people with motor impairments, some screen reader users, and a substantial number of users without a disability who happen to be navigating without a pointer. A defect in a live region affects screen reader users only. Neither ranking is more important in the abstract; they are different populations, and a queue that ranks them against each other needs a policy rather than a formula.

Keeping evidence from decaying

Every artifact in the pile has a shelf life. A scan result is valid for the build it ran against. A focus sequence is valid for the markup that produced it. A manual finding is valid until the component is redesigned. A conformance claim is valid for the version of the site it describes, and sites change continuously.

The practical arrangement ties each artifact to a trigger for re-evaluation. Scans re-run on every build, so their staleness is bounded by the build cadence. Sequence and widget tests re-run on the same cadence where they are automated, and on the release cadence where they are not. Manual findings are attached to the template and to the component, so a change to that component invalidates the finding. The exception list is reviewed on a schedule, because the reasons for exceptions expire: a third-party widget is replaced, a background image is removed, a browser bug is fixed.

The claim itself is the slowest-moving artifact and the one with the most inertia. A statement published on a site and not updated after a redesign is worse than no statement, because it asserts something false. The W3C’s statement generator exists partly to structure the claim so that its date, scope, and feedback mechanism are explicit, which makes staleness visible.

What the assembled evidence supports

At the end, the claim that can be made is narrow and precise: a named set of templates and states, evaluated at a named guidelines version and level, with a named sample ratio, by a named combination of automated and manual methods, with a named exception list. That sentence is checkable, and its weaknesses are visible in its own wording.

The temptation at this point is to compress it into a score. Scores are aggregations with weights that are not stated, over criteria that are not commensurable, and they move when the rule set moves. The distinction between what a machine can decide and what a person must is the reason: a score collapses the two classes into one number and hides the boundary, which is the single most important thing about the evidence.

What remains outside every claim is the experience of using the site. Conformance is a floor defined by testable criteria, and a site can conform at AA while being laborious to navigate with a screen reader, because the criteria do not measure the cost of a task, only the presence of the mechanisms that make it possible. The evidence described here establishes the floor and makes its limits explicit; the manual verification that goes beyond it is a separate activity with a separate, and necessarily smaller, sample.