Data Shape Debt and the Migration That Outlives the Code

Code debt is bounded by the next deployment. A bad function can be replaced in an afternoon, and the only trace of the old version is in history. Data debt is not bounded that way. Every record written under a bad shape carries it forward, every backup contains it, and every downstream consumer that read the bad shape has an expectation built on it. A design mistake in a schema is paid by migrations that run for months and by a period during which two shapes coexist and must both be correct.

The asymmetry has a specific cause: code is replaced atomically and data is not. A deployment switches every running instance to the new code within minutes, but the rows written before that moment still hold the old representation, and they will do so until something rewrites them. During the rewriting, both representations exist simultaneously, and both the old and new code may be running, because a rolling deployment does not switch everything at once and a rollback must remain possible.

The physical reason a schema change is expensive

On a relational store, most structural changes are not metadata edits. Changing a column’s type usually rewrites the table; adding a constraint usually validates every existing row; creating an index scans the table, and the scan plus the writes can hold locks that block modifications for its duration. In PostgreSQL the ordinary ALTER TABLE path takes an ACCESS EXCLUSIVE lock, which excludes both readers and writers for the length of the operation. Some alterations are metadata-only and effectively instant — adding a nullable column with no default is one — but distinguishing those from the rewriting ones requires knowing the specific operation, and the difference between “instant” and “an hour of downtime” turns on that knowledge.

The second cost is write amplification. A rewrite produces new row versions for every row, which in a system with multi-version concurrency control means the old versions remain until vacuumed, and the write-ahead log carries the full volume to replicas. A table of a hundred million rows rewritten in one transaction produces a WAL stream that replicas must replay, and replication lag during the rewrite is often the constraint that decides whether the change is feasible at all.

The third cost is human and is usually the largest: the change cannot be made in one step, so it must be made in an order that keeps every intermediate state correct for both old and new readers. That ordering is the whole content of the expand-contract discipline.

Expand, migrate, contract

The safe sequence has five phases, and each one must be compatible with the code version that precedes it.

Expand. Add the new structure without removing the old: a nullable column, a new table, a new index built with CREATE INDEX CONCURRENTLY so it does not block writes (and cannot run inside a transaction block, which forces it into its own migration step). Nothing reads the new structure yet.

Dual write. Deploy code that writes both representations and reads the old one. Both must be written in the same transaction, or the system has two truths that can diverge silently — the failure examined in Divergent Duplicates and the Cost of Two Truths.

Backfill. Populate the new structure for existing rows in bounded batches, idempotently and resumably. A single UPDATE over the whole table is one long transaction, and its cost in locks, WAL, and vacuum debt is paid all at once. Batches of a few thousand rows keyed by primary range keep each transaction short, and a rate limit keeps replication lag inside its budget. The job must be safe to rerun, because it will be interrupted.

Switch reads. Deploy code that reads the new structure, with a fallback to the old for rows the backfill has not reached. Verify by comparison: run both reads for a sample and count mismatches before trusting the new one.

Contract. Stop writing the old structure, then drop it once no code reads it and a rollback window has passed. This phase is where the debt is actually repaid, and it is the phase most often skipped.

-- expand: nullable, no default, so the change is metadata-only
ALTER TABLE orders ADD COLUMN total_minor BIGINT;

-- backfill in bounded batches, resumable by construction
UPDATE orders SET total_minor = ROUND(total::numeric * 100)
WHERE id > 1000000 AND id <= 1002000 AND total_minor IS NULL;

-- contract, only after the old column has no readers
ALTER TABLE orders DROP COLUMN total;

The money case, and why the shape matters

Storing money as a binary floating-point number is the canonical example of a shape that is cheap to write and expensive to undo. IEEE 754 binary64 cannot represent 0.1 exactly; the nearest value is slightly above it. Summing 0.1 ten times in binary64 yields a value that is not exactly 1, and comparisons that assume exactness fail at the boundary. The error is tiny per operation and does not stay tiny when values are summed over many rows or compared for equality after different orderings of the same additions.

The correct shape is an integer in the smallest currency unit, with the exponent stored separately or fixed by currency. Migrating to it is not a type change; it is a reinterpretation of every stored value, plus a decision about what to do with values that were already rounded under the old representation. The old values cannot be recovered exactly — the original decimal was lost at write time — so the migration usually has to accept the rounded value as the new truth and record that decision somewhere.

This is the general form of shape debt: the information needed to migrate correctly was destroyed by the original design. A column that stored a person’s full name as one string cannot be split reliably, because the boundary between given and family name is not encoded in the string. A timestamp stored without a timezone cannot be interpreted unambiguously afterwards, because the offset depended on a jurisdiction that was never recorded. A boolean that meant three things depending on a companion flag cannot be decomposed without reading the code that wrote it.

Where half-finished migrations accumulate

The expand phase is cheap and the contract phase is not, so migrations tend to stop in the middle. Three failure patterns follow.

Permanent duality. Both columns exist, both are written, one is read. New code was written against the new column and old jobs still read the old one, and nobody can say which is authoritative for a given row. The invariant that both agree is enforced by hope. Repair requires a comparison job, not a code change.

Nullable everything. The new column is nullable and stays nullable forever, because making it NOT NULL requires a validation pass and a lock, and the migration is “done”. The nullable column then forces every reader to handle a case that should be impossible, and the reader that forgets produces a null-pointer failure in production. A constraint deferred indefinitely is a permanent tax on every consumer of the data.

Orphaned artifacts. Shadow tables, comparison jobs, and feature flags from an abandoned migration remain, each consuming space and attention. The flag is the worst of these, because a flag whose two branches are both “live” in different environments doubles the state space for every test, as noted in Change Amplification: How Coupling Sets the Price of an Edit.

A migration record with four fields prevents most of this: the intended shape, the phase currently reached, the check that proves the phase is complete, and the removal step for each artifact created. The record is not bureaucracy; without it, the only person who knows whether the backfill finished is whoever ran it.

Events and the shapes that never die

When the system’s data flow is through durable messages rather than a shared database, a different version of the problem appears: the stored payloads cannot be rewritten at all, because they are consumed by readers that may replay history. A log retains every message in the shape it was written, and a new consumer reading from the beginning must handle every historical shape.

The standard resolutions are versioned event types and upcasting on read. Versioning by type name means a new shape is a new type, and consumers subscribe to the types they understand. Upcasting means the consumer holds a chain of transformations from each historical shape to the current one, applied at read time. Upcasting keeps the log immutable and pushes the compatibility code into the consumer, where it must be maintained forever for as long as the retention window extends — which, for a log retained indefinitely, is a permanent obligation.

The design constraint that follows is about what goes into the event. A payload that captures a full current-state snapshot binds the event to the schema of the day. A payload that captures the minimal fact that changed, with an identifier, is far cheaper to keep compatible, at the cost of requiring consumers to join. Neither is universally right; the choice determines how much of the schema becomes immutable.

Migration as a scheduled expense

The productive framing is that a data migration is a project with a start, a verification step, and an end, and that the end must be planned before the start. Without the end state written down, the migration’s artifacts become permanent structure, and the debt converts from a known principal into an open-ended interest charged on every reader, every backup, and every new engineer who has to learn which of two columns is real.

The scheduling question — when to spend the capacity, and whether to do it before or after the code-level repayments — is the general problem addressed in Sequencing Repayment Under a Fixed Capacity. The specific constraint that distinguishes data work is that it cannot be done in a quiet week later. Every day the old shape persists, more rows are written in it, and the backfill gets larger.