Schema Drift at the Connector Boundary

A connector is a translation layer between two schemas that change independently. The source’s schema changes when an application ships a migration; the sink’s changes when a consumer needs a new field. Neither change is coordinated with the other, and the connector sits between them holding a mapping that was correct at design time.

The failures that result are not uniform. Where the connector breaks depends on when it learns about the change, and the three timings have different blast radii.

Read-time: the query no longer matches

A polling connector that issues SELECT id, email, updated_at against a table that has gained a column still works, because the query names its columns. The new column is simply not captured, and the omission is silent. A connector that issues SELECT * fails differently: it receives a column it does not know how to map, and either drops it silently, errors, or writes it into a generic overflow column.

The SELECT * form is the more dangerous of the two, because it fails at a moment determined by the source’s migration rather than by the connector’s deployment. A migration that adds a nullable column changes the result set’s shape for every subsequent query, and a connector that assumed a fixed arity sees a mismatch mid-run.

Naming columns explicitly makes the omission visible instead: the connector captures a stable projection, and a new source column is missing from the sink until someone updates the projection. That is a gap in coverage rather than a crash, and it is detectable by comparing the source’s catalog against the connector’s configured column list, which is a cheap check to run periodically.

Decode-time: the log no longer matches the catalog

A log-based connector does not query, so it cannot name a projection. It reads row images from the log and decodes them against the table’s catalog as of the record’s position. A schema change in the source produces records in two formats: those written before the change and those after. The decoder must handle both.

Engines handle this by versioning the catalog alongside the log, so a decoder reading position $P$ uses the catalog that was current at $P$. PostgreSQL’s logical decoding maintains a catalog snapshot history for exactly this reason, and decoding an old position costs more than decoding a recent one because the older catalog must be reconstructed.

The failure mode when this is absent: a decoder applies the current catalog to a record written under an earlier one, and the column offsets do not line up. The result is either a decode error, which halts the connector, or a silently misaligned row in which the value that was in column 4 is written to column 5. The second is far worse, because it is plausible data.

A related hazard is the row image’s column count. A record written before a column was added contains fewer values, and a decoder that assumes the current column count either pads with nulls, which is correct if the column was added with a default that the log records and wrong otherwise, or it fails.

Write-time: the sink rejects the widened type

Even when the connector reads the new value correctly, the sink may not accept it. A source column that changes from VARCHAR(64) to VARCHAR(255) is the same type in most mappings, but a column that changes from INT to BIGINT may not be: a sink that declared a 32-bit integer column will overflow on values above $2^{31}-1$, and the failure appears as a write error on the specific rows that exceed the range, not on the schema change.

The failure is data-dependent and therefore intermittent, which makes it hard to attribute. A pipeline that runs for months and then fails on a handful of rows is exhibiting a type-width mismatch that was latent since the source’s migration.

Widening is generally safe and narrowing is not, and the asymmetry is worth stating as a rule: a sink column must be at least as wide as the source’s maximum, and the connector’s type mapping must be monotone under widening. Mappings that violate this, for example one that chooses the sink type from the source’s declared precision at connector creation time, are the ones that break later.

Numeric types are where the mappings diverge

The interesting differences between source and sink are not in strings and integers but in the types whose semantics differ across systems.

Decimals. A source NUMERIC(38, 10) has an exact decimal representation. A sink that stores it as a double-precision float loses exactness above 15 or so significant digits, and the loss is silent. Storing it as a string preserves exactness but loses the ability to sum it without a cast. Storing it as a decimal type preserves both, if the sink has one with enough precision.

Timestamps. The distinction between an instant and a local wall-clock time is the one that causes the most damage. A source column of type timestamp with time zone is an instant; a column of type timestamp is a wall-clock reading with no zone. A connector that maps both to the same sink type collapses the distinction, and a value that was written as local midnight in one zone becomes local midnight in another after a conversion. The Avro and Protobuf ecosystems both distinguish these, Protobuf’s well-known types including a timestamp for instants and no built-in type for local times, which is itself a signal about which one is meant to be portable.

Enumerations. A source enum with three values becomes a sink enum with three values, and a source migration adding a fourth breaks the sink’s constraint. Mapping enums to strings at the connector boundary avoids this at the cost of losing the constraint, and it is the mapping most connectors choose.

Arrays and nested structures. A source column holding a JSON document has no schema at the connector’s boundary. The connector can pass it through as text, in which case the sink’s consumers must parse it; or it can flatten it, in which case a change to the document’s shape breaks the flattening. Neither is wrong, and the choice determines where a later schema change surfaces.

Nullability. A source column that was nullable and becomes non-nullable is a tightening that the sink’s schema may not reflect, and a sink column declared nullable accepts nulls the source no longer produces. The reverse case, a non-nullable column becoming nullable, breaks a sink that has a NOT NULL constraint, and the failure appears on the first row that carries a null.

The schema registry pattern and what it enforces

Systems that carry schema metadata alongside the data, whether a schema registry for a log or a catalog for a table format, can enforce compatibility rules at write time rather than discovering incompatibility at read time. The rules are stated as pairs:

• Backward compatible: a reader using the new schema can read data written with the old one. Adding a field with a default is backward compatible; removing a required field is not.
• Forward compatible: a reader using the old schema can read data written with the new one. Adding a field is forward compatible if the reader ignores unknown fields; removing one is not.
• Full: both directions hold.

The definitions are about the reader’s schema relative to the writer’s, and the enforcement is mechanical: the registry compares the proposed schema against the previous one and rejects the registration if the configured rule fails. What it cannot enforce is semantic compatibility. A field that changes from storing cents to storing dollars has the same type and is fully compatible by any structural rule, and every consumer’s arithmetic is now wrong. Schema rules catch structural breaks and are blind to meaning.

Versioning the connector’s mapping

The connector’s own mapping is a schema in its own right, and it needs a version. When the source adds a column, the connector’s mapping gains an entry, and the records it emits after that point have a different shape from those before.

If the emitted records carry a schema identifier, consumers can handle the transition. If they do not, the consumer discovers the change by encountering a field it does not expect or missing one it does, and the discovery happens at an arbitrary record.

The practical discipline is that the connector emits a schema version with each batch, and the sink stores it. A batch written under version 7 and a batch written under version 8 are then distinguishable, and a query over a range spanning both can decide how to interpret each. Without that, the sink’s history is a mixture of shapes with no record of where the boundary is, which makes a later backfill the only way to make the history uniform.

Defaults and the historical rows

A source migration that adds a column with a default value gives existing rows the default at the moment of the migration, and new rows the default unless specified. From the log’s perspective, the migration may or may not rewrite the existing rows depending on the engine’s implementation: some engines store the default in the catalog and apply it at read time without touching the rows, so the log contains no record of the change to existing rows.

That distinction matters for a capture stream. If the default is applied at read time, a connector that reads rows sees the default on old rows, while a connector that reads the log sees no change event for them. The two capture paths disagree about whether the historical rows have the new column, and a sink populated by the log alone will have nulls where the source shows the default.

The resolution is either to accept the divergence and treat the sink’s null as “not recorded” rather than “absent”, or to run a backfill that reads the rows and writes the default. The first is cheaper and requires every consumer to understand the distinction.

Detecting drift before it breaks

The checks that catch drift are cheap and are worth running on a schedule rather than after a failure.

A comparison of the source’s column metadata against the connector’s configured projection catches added and removed columns. A comparison of declared types catches widening that the mapping cannot absorb. A comparison of the sink’s constraints against the source’s nullability catches the cases where a write will fail on a specific row.

The third check is the one usually omitted, because it requires reading the sink’s schema rather than the source’s. It is also the one that predicts a runtime failure most reliably, since a constraint violation is a write error rather than a silent misalignment.

The remaining class, a value whose meaning changed without its type changing, is not detectable by any structural check, and is the reason reconciliation compares values rather than counts.