Mixed-Version Operation: What Two Versions Must Agree On

A rolling upgrade is not a replacement of one version by another. It is a period during which both versions run simultaneously, exchanging traffic, reading and writing the same data, and each believing it is correct. The length of that period is the sum of the wave soaks and the time to replace the last instance, and every interface between the two builds must remain valid for its whole duration. Compatibility work is therefore not a review checklist applied to a diff; it is a constraint on what a single deployment is allowed to change.

The window is longer than the deployment

Operators tend to measure a rollout by the time the controller takes to finish, but the exposure window is bounded by different events. It opens when the first instance of the new version begins serving, and it closes only when the last instance of the old version stops. Anything that outlives instance replacement extends the window past the rollout’s own end: messages queued before the upgrade that are consumed after it, scheduled jobs that run once per hour, cached entries with a time-to-live measured in hours, and client applications that the operator does not control at all.

That last category is the one that breaks the model. A service with a public API has a mixed-version population consisting of every client build in use, and the operator cannot replace those instances. The compatibility window is then not measured in minutes but in the deprecation policy of the API, months or years. The same reasoning applies to on-disk formats: a new build that writes a format the previous build cannot read makes rollback impossible from the moment the first new instance writes, even if every instance is still healthy.

The practical rule that follows is that a deployment may only introduce changes that are readable by the version being replaced. This is the difference between a change being correct and a change being deployable, and it is why the sequence of changes for any nontrivial modification is spread across multiple releases rather than shipped in one.

Expand, migrate, contract

The standard shape for a change that alters a shared format has three phases, deliberately separated into three deployments.

Expand. Add the new representation without removing the old one. A new column is added as nullable; a new message field is added with a default; a new index is built alongside the existing one; a new topic is created while the old one keeps receiving writes. Both versions of the code can now run: the old version ignores what it does not know, the new version tolerates the absence of what it expects.

Migrate. Fill in the new representation for existing data and begin writing both. A backfill populates the new column, a dual-write path populates both fields, a reconciliation job copies old records into the new structure. During this phase both representations are authoritative, and a disagreement between them is a bug that must be detectable, which is why the migration usually includes a verification pass that compares the two rather than trusting the write path.

Contract. Remove the old representation once no reader depends on it. Drop the column, remove the field from the schema with its number reserved, delete the old index, stop the dual write. This phase can only happen after every instance of the previous version has been replaced and after any queued or delayed consumer has drained.

The phases cannot be collapsed without losing the ability to roll back. A deployment that adds a column and starts writing it in the same release is safe only if the old code tolerates the column’s presence, which it does; the danger appears in the reverse direction, when the new code stops writing the old column and the old code is still running. That is a contract step executed before its precondition, and it makes the previous version unable to read data written after the upgrade.

-- Expand: nullable, no default rewrite, safe for old readers.
ALTER TABLE orders ADD COLUMN region_code text;

-- Migrate: backfill in bounded batches so the table stays writable.
UPDATE orders SET region_code = derive_region(shipping_address)
WHERE region_code IS NULL AND id > $1 AND id <= $2;

-- Contract: only after no running version reads the old column.
ALTER TABLE orders DROP COLUMN shipping_region;

The ALTER TABLE ... ADD COLUMN form matters on large tables. Adding a nullable column with no default is a metadata-only operation in the common relational engines; adding one with a non-null default may rewrite the table or require a separate backfill, depending on the engine and version. A migration that takes an exclusive lock for the duration of a rewrite blocks writes for as long as the rewrite takes, which on a table of tens of millions of rows is measured in minutes. The expansion phase exists precisely to keep that work off the critical path.

The interfaces that must be bidirectional

Four categories of interface carry the compatibility requirement, and each fails differently.

Request and response schemas. A new field added to a response is safe if old clients ignore unknown fields, which most serialisation formats do by default and some do not. A new required field in a request is a breaking change the moment any old client calls. Field numbers, not names, are the identity in protobuf-style encodings: reusing a retired field number for a different meaning silently corrupts data written by an old producer, which is why the numbers of removed fields are reserved rather than recycled.

Persisted records. A row written by the new version must be readable by the old version if rollback is to remain possible. This constrains the direction of change: new columns must be nullable or defaulted, new enum values must be treated as unknown rather than rejected, and a format version field must be read by both sides with an agreed interpretation. Adding an enum value is a classic trap: the new writer emits it, the old reader has no case for it, and the behaviour depends on whether the decoder errors, maps to a default, or preserves the unknown value.

Message queues. A queue decouples production from consumption in time as well as in space, so messages written by the old version may be consumed by the new version and vice versa, possibly long after the rollout. A consumer that assumes every message has a field introduced in this release will fail on the backlog. The safe pattern is for consumers to tolerate the absence of new fields and for producers to keep emitting fields that old consumers still require, which means the producer cannot drop a field until the consumer population has fully turned over.

Protocols between services. Two services upgraded independently produce four combinations of versions, and all four must work during the overlap. A protocol negotiation that lets each side advertise its capabilities reduces the matrix, but only if the negotiation itself is compatible in both directions: a handshake that the old version does not understand is a breaking change regardless of what it negotiates.

Detecting the cases that were missed

The failure signature of a missed compatibility case is distinctive: the rollout proceeds normally through the early waves and fails when the two versions begin to interact. Errors appear as decode failures on data written moments earlier, as 400 responses from a client that was working before, or as a consumer that stalls on a message it cannot parse. Because the early waves may contain only new instances talking to each other, the canary can pass cleanly and the failure appears at the wave where old and new are mixed in comparable proportion.

Testing for it requires running the mixed state deliberately. A pre-merge check that starts one instance of the current build and one of the candidate build against a shared database and a shared queue exercises the actual matrix, including the direction that is easy to forget: new code reading old data, and old code reading new data. The second direction is the one that determines whether rollback works, and it is rarely covered by tests written alongside the new feature.

The recovery path matters as much as the detection. A rollout that fails a compatibility check must be able to stop and revert without leaving data that the old version cannot read. That is exactly the property the expand phase preserves: as long as no contract step has run, the old version can still operate on the data, and the rollback is a redeploy. Once the contract step has run, the rollback path is gone and the only remaining option is to roll forward, which is why the decision to make a change reversible at all belongs before the migration is scheduled, not after it fails.

Cost of the discipline

Three releases instead of one, three rollout windows, and a period during which the system writes data twice or stores it redundantly. That is the price of keeping the previous version viable throughout. The alternative is a single release that changes format and code together, which works whenever the change is genuinely reversible (a pure computation change with no persisted state) and fails whenever it is not.

The useful way to decide which applies is to ask what the previous version would do with data written by the new one. If the answer is “read it correctly”, the change can ship in one release. If the answer is “misread it”, “crash”, or “silently drop it”, the change needs the three-phase treatment, and the rollout plan needs to reflect that the phases are separate deployments with their own waves and gates, not steps inside one.