Reference Implementations and the Conformance Laboratory

A specification states what conforming behaviour is. A conformance suite states what somebody decided to check. The distance between the two is where most interoperability failures originate, and the industry’s main instrument for closing it is not the suite but the second implementation — a program written from the text by people who did not have the first program’s source.

The Open Systems Interconnection work of the 1980s and 1990s generalized this into an explicit methodology, published as ISO/IEC 9646, with named test types: abstract test suites derived from the specification, executable test suites derived from the abstract ones, and a verdict assigned per test case. The methodology was formal and the technology it described largely did not survive, but the vocabulary did, and it names the problem precisely.

Why one implementation is not evidence

A single implementation is compatible with any number of incompatible readings of a document. It cannot distinguish “the specification says this” from “the authors assumed this”. Every ambiguity in the text is silently resolved to the author’s preference, and the resulting product is not a test of the specification at all — it is the specification’s first draft of the parts that were left open.

A second implementation written against the document alone functions as a differential probe. Where the two implementations differ, either the specification is ambiguous or one of them is wrong. Discriminating between those cases requires going back to the text, and doing so forces the text to be read closely for the first time. This is why the useful output of an interoperability exercise is usually a set of specification defects, not a bug list for either implementation.

Histories of protocol standardization are full of this pattern: implementers assemble, run messages between independent stacks, and spend the session discovering that a field both sides believed to be mandatory is described in two places with different widths. The remedy is an erratum or an implementation note; the diagnosis is only available because two independent readings existed to compare.

Abstract tests, executable tests, and the derivation gap

The ISO/IEC 9646 framework separates three layers, and the separation is the point.

An abstract test suite is derived from the specification and states, for each behaviour, a test purpose and a test case expressed in some implementation-independent notation. An executable test suite is a realization of the abstract suite for a particular system under test, with concrete messages, timing, and access methods. A verdict is then the result of running the executable suite: pass, fail, or inconclusive.

Two derivation gaps appear here and neither is eliminable. The first is between the specification and the abstract suite: a behaviour that the specification requires but the suite’s authors did not think of is unreachable by any test. The second is between the abstract suite and the executable one: the test harness’s own assumptions about the system under test — how to observe internal state, how to inject malformed input — constrain what can actually be checked.

The practical consequence is that passing a conformance suite proves that the implementation satisfies the tested subset. It says nothing about the untested remainder, which for a mature specification is the larger part. Certification bodies mitigate this with a required test suite that grows over time, and with periodic reviews that add test cases for defects discovered in the field. The growth of such a suite is itself a record of ambiguity.

Writing the test that makes the clause testable

Concrete clauses are cheap to test. Vague ones are not, and a suite author facing a vague clause has only two honest options: skip the clause, or pick a reading and label the test as one reading.

A clause of the form “the receiver SHOULD reject a frame whose checksum does not verify” can be tested only if the suite declares how strongly it enforces SHOULD. Most suites choose to treat SHOULD violations as warnings and MUST violations as failures, which means the suite’s verdict encodes a policy about the specification’s middle tier. That policy is not in the standard and is not derivable from it.

For an implementation author, the suite’s policy is worth knowing independently of the specification. A verdict of “pass with warnings” may correspond to a product that is deliberately non-conforming for a stated reason, and the reason is often more interesting than the warning.

Certification as a market instrument

Conformance certification exists because procurement needs a discriminator. A purchaser who cannot evaluate two products technically will accept a certificate, and a certificate requires a process with a laboratory, a test report, and a party willing to sign.

The instrument has well-known pathologies. Certification raises the cost of entry, which favours incumbents able to absorb the fee and delay. It freezes the tested subset at whatever the suite covered when the lab was accredited, which can lag the specification’s evolution. And it can validate an implementation’s behaviour against the suite rather than against peers, so that two certified products fail to interoperate when both followed different readings of an untested clause.

Registration, as distinct from certification, is the weaker instrument: the implementer asserts conformance and makes a declaration, and the body publishes the declaration without running tests. Registration is cheap and covers the long tail of small products; it also means the published list of conforming implementations is a list of claims. Both instruments answer a procurement question — who is accountable if this fails — more directly than they answer the engineering question of whether two systems interoperate.

The Open Group’s POSIX certification is the instructive case, because the standard is unusually large and its testing necessarily partial. A POSIX-conformant system is certified as passing a defined suite; the fraction of the full system interface that the suite exercises is a small share, and applications that rely on behaviour outside it discover the difference at runtime rather than at certification. Systems that pass certification and still behave differently under specific calls are the normal state of the world rather than a scandal.

The reference implementation as a fallback authority

Where a formal suite does not exist, the de facto reference implementation does the same job less formally. Its behaviour becomes the test. Two failure modes follow.

First, the reference acquires accidental obligations: behaviour nobody intended to standardize becomes required because clients depend on it, and any revision of the reference that changes an unstandardized behaviour breaks compatibility without a specification change to point at. Second, the reference’s bugs become the specification. A defect that several independent implementations reproduce because they were written by reading the reference’s source is not distinguishable, downstream, from an intended feature.

Mitigations are procedural rather than technical. Keeping the reference implementation’s source unavailable to the second implementers is a real practice and its purpose is exactly this: to force the second team through the document. A specification’s normative text plus a differential exercise between implementations that were not sharing source is the strongest evidence of meaning available. The documents that canonicalise that evidence, and the mechanism by which they accumulate corrections, are covered in the account of errata and withdrawal. The exchange of committee positions that produces the text is described in the account of voting rules and objections.

Conformance laboratories do not make specifications correct. They locate the places where the document did not commit, and they convert those places into either errata or precedent. Which of the two a given ambiguity becomes is decided by who filed the test case first.