Evolving a Persistent Domain Model Without Downtime

Renaming a type in code is a mechanical refactor that a compiler verifies. Renaming the concept it represents is not, because the old name is written into rows, indexes, event payloads, and API responses that other systems have already consumed. The code can be changed in one commit; the data changes over hours or weeks, while both the old and the new shapes are live.

The data is the expensive part

Every model change has three surfaces. The in-memory types are cheap to change because the compiler finds the call sites. The stored representation is expensive because it is large, must be migrated while serving traffic, and may be read by code that will not be redeployed at the same moment. The published representation — events and API payloads — is the most expensive, because the consumers are outside the change’s control entirely.

The asymmetry has a practical consequence: the design goal is not to avoid changing the model but to make each change decomposable into steps that are individually safe. A step is safe if old code and new code can both run against the resulting schema, and if running the old code after the new code has started is not catastrophic. Those two conditions are what a rolling deploy requires, and they rule out any change that reinterprets an existing column in place.

A rename illustrates the problem. Changing a column’s meaning while keeping its name means old code reads new data with the old interpretation. Changing the name means old code fails with a column-not-found error the moment the new schema is applied. Both are single-step changes that cannot be rolled out gradually.

Expand, migrate, contract

The standard sequence breaks the change into three deploys, each of which is safe on its own.

Expand adds the new structure without removing the old. A new column is nullable, or has a default; a new table exists alongside the old one; a new event field is added and old consumers ignore it. The application at this stage may not use the new structure at all, or may write to both.

Migrate moves the data and the reads. A backfill copies existing rows into the new shape. The application is deployed to write both shapes and read the new one, with a fallback to the old for rows the backfill has not reached. Once the backfill is complete and verified, a further deploy stops writing the old shape.

Contract removes the old structure: drop the column, delete the table, remove the fallback code, remove the dual-write. This deploy is the only one that is not reversible in a data sense, so it comes last, after a period in which the old structure was written but never read.

The sequence is longer than the change, and that is the cost. Three deploys, a period of dual writes, and temporary code that must be tracked and removed. The alternative — a single coordinated change — requires stopping writes, migrating, and restarting, which is fine for a system with a maintenance window and impossible for one without.

Dual writes deserve suspicion. Writing both shapes in the application is not atomic with respect to a reader that reads only one, and if the second write fails the two diverge silently. Where possible, the old shape should be derived from the new by a trigger or a generated column, so there is one source of truth. Where a trigger is unavailable, the dual write must be inside the same transaction as the primary write, and a reconciliation job must check for divergence.

Backfills that do not lock the table

A single UPDATE over a large table takes row locks for the duration, bloats the write-ahead log, and can block replication. The practical approach is a batched loop with a bound on each batch and a pause between batches.

-- One batch: copy the old column into the new one for a bounded set of rows.
WITH batch AS (
  SELECT id, legacy_code
    FROM shipment
   WHERE new_code IS NULL
   ORDER BY id
   LIMIT 1000
   FOR UPDATE SKIP LOCKED
)
UPDATE shipment s
   SET new_code = batch.legacy_code
  FROM batch
 WHERE s.id = batch.id;

FOR UPDATE SKIP LOCKED lets several workers run concurrently without contending on the same rows, and the WHERE new_code IS NULL predicate makes the loop resumable after a crash: a worker that dies mid-batch leaves the unprocessed rows in the same state. Each batch commits separately, so a failure loses at most one batch.

Three properties need watching during a backfill. Replication lag grows with write volume, so the batch size is tuned against the replica’s capacity rather than the primary’s. A backfill that runs against rows being updated concurrently needs a rule for which version wins — usually the application is the authority, and the backfill must skip rows modified after it started. And the backfill’s progress must be observable, because “almost done” is not a state anyone can act on.

The same pattern applies to a table split or a value normalisation. Splitting a table means writing to both, backfilling the new table from the old, switching reads, then stopping writes to the old. Normalising a free-text field into a lookup table means populating the lookup, backfilling the foreign key, switching reads, then dropping the text column. The shape varies; the phases do not.

Event schemas age too

Published events are append-only history, so a change to an event’s shape leaves old events in the old shape forever. A consumer that reads the stream must handle every version it might encounter, and the versions accumulate.

Two mechanisms keep this manageable. The first is additive change: new fields are optional, consumers ignore unknown fields, and old events remain valid because they simply lack the new field. This works only when the new field can be absent without changing the meaning of the event, which excludes most changes of interpretation.

The second is upcasting: a transformation applied on read that converts an old event version into the current one before the consumer sees it. The upcaster for version 1 to version 2 might rename a field, split an amount into an amount and a currency, or discard a field that has become derived. Each event carries a version number, and the consumer’s deserialiser walks the chain from the stored version to the current one.

type StoredEvent = { type: string; version: number; payload: Record<string, unknown> };

const upcasters: Record<string, (e: StoredEvent) => StoredEvent> = {
  "OrderPlaced:1": (e) => {
    const { total, ...rest } = e.payload;
    return {
      ...e,
      version: 2,
      payload: {
        ...rest,
        // Version 1 stored the total as a bare number; version 2 pairs it with a
        // currency that was implicit in the original single-currency system.
        amount: { minorUnits: total, currency: "EUR" },
      },
    };
  },
};

function toCurrent(e: StoredEvent): StoredEvent {
  let current = e;
  while (`${current.type}:${current.version}` in upcasters) {
    current = upcasters[`${current.type}:${current.version}`](current);
  }
  return current;
}

Upcasting keeps replayed history readable without rewriting stored events, which matters because rewriting an append-only log undermines the reason for keeping it. Its cost is that the upcaster chain is code that must live as long as the oldest retained event, and it must be tested against real historical payloads rather than constructed examples, because the surprises are in the fields that were never documented.

Where events have been stored without a version number, the first evolution is a detective exercise: the shape must be inferred from timestamps and payload keys, and events may differ within the same nominal type. The absence of a version field is the single most expensive omission in an event schema.

Rollback is a first-class requirement

Every deploy in the sequence may need to be reverted, and the data written by the new code will still be there. A rollback that restores the old code must therefore find the old shape still populated and still correct. This is why dual writes continue until the contract phase and why the old column is not dropped until the new one has been the only source of truth for a full deploy cycle.

The harder rollback is the one that involves a semantic change. If the new code wrote values that the old code cannot interpret, reverting the code leaves unreadable rows. The mitigation is to keep the old shape’s values in the old units and the new shape’s values in the new units during the migration window, with an explicit conversion at the boundary, rather than converting in place and hoping.

Model changes that alter invariants are the worst case. If the new model forbids a state the old model allowed, rows in that state must be resolved before the new code can run, and the resolution is a business decision, not a migration script. The narrowing techniques in Invariants and Illegal States state what the new code requires; the migration has to find every row that violates it and either fix it or route it to a human. Running that query before deploying is the difference between a rollout and an outage.

Finally, deletions. Removing a concept from the model is a contract-phase change with no expand counterpart, and it can destroy data that later turns out to be needed. The safe form is to stop reading first, keep writing for a full cycle, and only then drop — and for records with any audit or reporting value, to mark them superseded rather than delete them, as discussed in Modeling Validity Over Time. The storage cost of retention is usually smaller than the cost of reconstructing history that no longer exists.