Breaking Change Detection in API Descriptions

A description of an HTTP API is a machine-readable statement of the interface, and a difference between two descriptions is a machine-readable statement of what changed. The temptation is to conclude that a diff is a compatibility verdict. It is not. A diff reports differences; whether a difference breaks a caller depends on things the description does not contain, and on which callers exist.

Tools that compare OpenAPI documents, such as oasdiff, and tools that compare Protobuf schemas, such as Buf’s breaking rules, encode a set of judgements about which differences matter. Those judgements are the useful part, and they are worth understanding rather than merely running.

Rules decidable from the description alone

Some changes are breaking under any reasonable consumer, and a description-level check catches them without ambiguity.

Removing an operation is the clearest case. Removing a path, or removing a method from a path, deletes an interface that some caller may be addressing. Adding a required request parameter or request body property is likewise breaking regardless of the payload’s shape, because every existing request that omits it becomes invalid. Removing an enum value is breaking for any caller that sends it; adding one is breaking for any caller that exhaustively switches on the response values, which is why enum additions belong to a warning category rather than a safe one.

Retyping a field is breaking when the types are not mutually parseable. Narrowing a numeric range, lowering a maximum length, or tightening a pattern rejects requests that previously passed. Changing a response status from 200 to 201 breaks clients that branch on success codes specifically rather than on the class.

# before
properties:
  total_cents:
    type: integer
---
# after
properties:
  total_cents:
    type: integer
    minimum: 0     # new constraint: rejects previously valid negative values

The minimum addition is invisible to a type comparison and obvious to a range comparison. A checker that only compares type and required will call this change compatible, and it is not.

Rules that need the wire format

For binary schemas, the format supplies rules that the description alone cannot. In Protobuf, changing a field’s number is a removal plus an addition, changing its type across wire-type families is incompatible, and removing a field without reserving its number allows a future reuse that silently corrupts data. A checker that parses the .proto files directly can decide all three, which is why schema-level tooling gives firmer answers than description-level tooling.

The same holds inside an OpenAPI document for constructs with defined semantics: oneOf to anyOf is a loosening that breaks consumers relying on exactly-one matching, and moving a property from the root of a request body into a nested object is a removal plus an addition at two different paths. These are decidable, but only by a checker that models the schema language rather than comparing JSON trees.

Descriptions that embed JSON Schema, as OpenAPI 3.1 does, inherit JSON Schema’s vocabulary, including additionalProperties, const, exclusiveMinimum, and conditional if/then constructs. A checker that does not model the vocabulary will miss the constraints expressed with it. That is the practical argument for using a tool built around the schema language rather than a generic structural diff, however convenient the generic diff is to write.

Where static analysis cannot reach

The most damaging changes are usually absent from the description. Three categories recur.

Semantic changes. The endpoint still returns 200 with a total_cents field, but the field now excludes tax. No diff sees it. The contract changed in the only sense that matters to an accounting integration, and the only defence is a human review of behavioural changes and a test suite that asserts on values rather than shapes.

Documentation and defaults. Changing the default value applied to an omitted parameter is a breaking change with no schema manifestation unless the default is declared in the description. Some APIs deliberately omit defaults from the document for this reason; others declare them and treat the declaration as a promise.

Infrastructure behaviour. Rate limits, timeouts, maximum page size enforcement, and the set of headers added by a gateway are rarely in the description and are frequently what a caller depends on. A page size that was silently capped at 100 and is now capped at 50 breaks a client that assumed full pages, and nothing in the document changed.

For these, the description diff is a floor rather than a ceiling. It catches the mechanical cases cheaply, which frees review attention for the ones that need judgement.

Severity levels and the noise problem

A checker that reports every difference as breaking is quickly ignored, because a diff includes legitimate additive changes. The workable classification has three levels: breaking changes that must not ship without a major version bump or a migration plan, warnings that are compatible today but risky, and informational changes.

The warning tier is where judgement concentrates. Adding an optional response property is informational for a tolerant reader and breaking for a strict one. Adding an enum value is informational for a caller that stores the string and breaking for one that switches on it. Marking a property as deprecated is informational in the description and a commitment in the process.

The classification has to match the API’s documented consumer behaviour, which is exactly the material in tolerant readers and unknown fields. A checker configured to treat additions as safe has made an assumption about its consumers, and that assumption belongs in the configuration with a comment, not in a default nobody examined.

Baseline selection

Comparing against the previous commit is cheap and mostly wrong: commits that do not change the interface produce an empty comparison, and the ones that do may be documentation fixes. The meaningful baseline is the description of the version that consumers are actually using, which is normally the last release on the current major version.

That baseline has two consequences. First, the check needs to run against a released artifact rather than a working tree, which means the description has to be versioned alongside the code and published somewhere retrievable. Second, the check needs to know which version is the current baseline, which is a policy question: the current major version’s latest release, not the latest release overall, because changes to an unreleased future version are not yet anybody’s contract.

For providers that support several major versions in parallel, the baseline is per-version, and the same change can be breaking for one and correct for another. Running the check once against a single baseline produces false alarms on the version where the change is intended.

Suppression and the drift toward green

Every checker eventually grows an ignore list. A finding that the team believes is safe gets suppressed, the check turns green, and the suppression outlives the reasoning that justified it.

Two disciplines limit the damage. Suppressions are narrow, keyed to a specific finding on a specific path rather than to a rule, so a new instance of the same mistake is still reported. And each suppression carries the reason and the date, so a reviewer can decide whether it still holds. A suppression file with dates is a lightweight record of deliberate decisions; a suppression file without them is a list of things nobody remembers choosing.

The alternative to suppression is adjusting the rule globally, which has the same effect on every future change and no record of why. Neither approach makes the unsafe change safe; both merely stop the check from complaining about it.

What the gate should do

A detection tool produces findings; the merge gate decides what to do with them. The useful arrangement blocks merges only on the breaking tier, requires an explicit acknowledgement for warnings, and records the full report as a build artifact so the reasoning is recoverable after the fact.

Where a breaking change is intended, the acknowledgement should be the version bump and the migration note rather than a checkbox. Tying the approval to an artifact that consumers will read means the mechanism that silences the checker is the same one that informs the people affected, and the two cannot drift apart. The policy for what happens after that point, including the announcement and the removal date, is the subject of deprecation windows and removal gates.

A description diff is a good instrument for one specific job: proving that a release did not change the mechanical shape of the interface. It is a poor instrument for proving that a release did not break anybody, because the information required for that judgement lives in the consumers, not in the document.