Where a Schema Contract Is Enforced, and What It Cannot Catch

A file arrives with a column of order identifiers. In one batch they are digits, in the next they carry a leading zero, and in a third a handful of rows hold the string N/A. Nothing in the file announces which of these is intended. A pipeline that infers types at read time will decide per file: integers here, strings there, and a null in the third. Three query plans later, a join between orders and shipments silently loses rows, because the join key has two types and the comparison between them is either an error or a coercion that drops the leading zeros.

The failure is not a parsing bug. It is the absence of a decision about where the type of a column is fixed. That decision, more than any validator library, is what a schema contract encodes.

Two places to bind a type

A schema can be applied when data is written or when it is read. The two arrangements produce different failure surfaces, and neither is strictly better.

Under write-time enforcement, the writer must produce records conforming to a declared schema, and the storage layer records that schema alongside the data. A record that does not conform is rejected, quarantined, or coerced at the boundary, before any downstream consumer can observe it. The cost is paid once per record and by the producer, who must now know the contract. The benefit is that every reader can rely on the type without inspecting anything.

Under read-time inference, the writer stores whatever it has, and the reader derives a schema from the bytes. A CSV reader samples rows, guesses whether a field is an integer or a string, and the answer can differ between two files in the same directory. JSON readers face the same problem in a milder form, since JSON distinguishes numbers, strings, booleans, and null syntactically but not integers from floats, nor dates from strings.

The practical difference is where the ambiguity lands. Write-time enforcement turns a downstream correctness problem into an upstream rejection, which is visible and attributable. Read-time inference turns it into a query that returns a plausible but wrong answer, which is neither.

A declared schema still leaves a large space unspecified. It constrains names, order, types, and which fields may be absent. It says nothing about whether status may hold a value outside a known set, whether quantity may be negative, whether end_date precedes start_date, or whether a foreign key resolves. Those are semantic constraints, and they require a different mechanism: either checks expressed against the data after it lands, or constraints the storage engine itself enforces, as a relational database does with CHECK and FOREIGN KEY. Conflating the two is the most common way a validation layer looks complete while letting the expensive errors through. Structural validation is cheap and total; semantic validation is expensive and always partial.

What a parser will do to a value without asking

Type coercion is where schema enforcement quietly stops being enforcement. A validator that accepts "42" for an integer field, or 42 for a string field, is not validating a type; it is normalising one. The distinction matters because coercion is lossy in one direction and not the other.

Widening conversions lose nothing: a 32-bit integer fits in a 64-bit integer, and a DATE fits in a TIMESTAMP. Narrowing conversions lose information: a decimal with scale 4 written into a column of scale 2 is rounded, and a timestamp with a time zone written into a zone-less column is either shifted to some assumed zone or stripped, depending on the writer. A string written into a timestamp column depends on a format that the reader has to guess, and 03/04/2025 is a valid date under at least two incompatible conventions.

The decimal case has a hard ceiling worth knowing: the fixed-length decimal encoding in Parquet supports at most 38 digits of precision, and the same limit propagates into table formats built on it. A value with more digits cannot be represented at all, so it is either rounded or rejected, and the choice is the writer’s.

Nullability is a second place where a schema says less than it appears to. Three states are routinely collapsed into one: the field is absent, the field is present and null, and the field is present and holds an empty string. A schema that declares a field optional permits all three, and a downstream COUNT of non-null values will treat them identically while a WHERE clause will not. Distinguishing them requires a reader that preserves the distinction, which is exactly what a typed columnar format does and what CSV cannot.

# Distinguishes three states that a naive null check collapses.
def check_row(row: dict, required: set[str], typed: dict[str, type]) -> list[str]:
    problems = []
    for name in required:
        if name not in row:
            problems.append(f"{name}: absent")
    for name, value in row.items():
        if value is None:
            if name in required:
                problems.append(f"{name}: explicit null in required field")
            continue
        expected = typed.get(name)
        # bool is a subclass of int in Python; a JSON true would otherwise pass as 1.
        if expected is int and isinstance(value, bool):
            problems.append(f"{name}: bool where int expected")
        elif expected is not None and not isinstance(value, expected):
            problems.append(f"{name}: {type(value).__name__} where {expected.__name__} expected")
    return problems

Names, identifiers, and the drift that renames

Types are only half of what a schema pins down. The other half is identity: which field in the new file is the same field as the one in the old file.

Two resolution strategies exist. Name-based resolution matches fields by their names, as Avro’s reader/writer schema resolution does, and fails when a producer renames a column. Identifier-based resolution assigns every field a stable integer ID at creation, as Parquet and Iceberg do, and matches by ID so that a rename is a metadata change rather than a data change. The second strategy tolerates renames and is strictly more expensive to maintain, because IDs must be assigned once and preserved forever, and any tool that rewrites a schema without carrying IDs forward breaks the guarantee.

Name-based resolution has a specific failure mode that is worse than an error. When a writer’s schema lacks a field the reader expects, and that field is optional in the reader’s schema, resolution supplies the field’s default and proceeds. Every row in the new partition now carries the default, and nothing in the query result indicates that the value was invented at read time rather than written. A rename on the producer side therefore appears as a column that suddenly holds a constant. Detecting it requires comparing per-partition null rates or default rates, not inspecting the schema, because the schema is consistent by construction.

Case sensitivity adds a smaller version of the same problem. Unquoted identifiers in SQL fold to a case determined by the engine, quoted ones do not, and CSV headers carry whatever the producer typed. Two files with OrderID and order_id can resolve to one column or two depending on the engine and the quoting, and no schema validator sees the conflict if it is applied per file.

Compatibility directions and what they permit

Schema registries describe change with a compatibility vocabulary, and the vocabulary is worth stating precisely because the words are used loosely.

A change is backward compatible if data written with the new schema can be read by a consumer using the old schema. Adding an optional field with a default is backward compatible: the old reader ignores the field. Removing a required field is not, because the old reader expects it.

A change is forward compatible if data written with the old schema can be read by a consumer using the new schema. Adding a required field without a default is not forward compatible: old records have no value for it and the new reader has nothing to substitute.

A change is fully compatible when both hold, which generally means only adding or removing optional fields. The asymmetry is the point: the direction of compatibility is named from the perspective of the reader, so the same change is backward compatible for new writers and forward compatible for new readers, and those are different claims about different populations of files.

The compatibility model also has a temporal hole. Compatibility is evaluated between consecutive schema versions, but a dataset is read across all versions at once. A field added in version 3 and removed in version 7 leaves partitions that have it and partitions that do not, and a query over the whole table must handle both. Columnar formats with field IDs handle this by treating absent fields as null, which is correct but hides the fact that the null is historical rather than factual. The distinction between “this row had no value” and “this row predates the column” is not representable, and any analysis that counts nulls as missing data will be wrong for the older partitions. That is one reason a null rate is only interpretable alongside the schema history of the partitions it was computed over.

Coercion also decides what the contract means, because a validator that normalises is a validator that accepts. The clearest case is a string field receiving an integer, or an integer field receiving a string that happens to parse: the record passes, the column’s type is whatever the reader chose, and two writers that make opposite choices produce a table that is consistent per file and inconsistent per partition. The same problem appears in reverse at read time, where deduplication on a key has to decide whether two records whose fields differ only in type are the same record.

The validation boundary and its cost

Where a check runs determines what it can prevent. A check inside the writer can refuse a record before it is durable, so nothing downstream ever sees it. A check after the write can only observe and route, and the bad record has already been committed to a partition that other consumers may read.

Running validation inside the writer requires the producer to carry the schema and to handle rejection, which is a real cost when the producer is a third-party application that cannot be modified. Running it immediately after the write, on the landed file, is cheaper to deploy and catches the same errors one commit later. Running it before a consumer reads is the weakest position: the check then protects only the queries that invoke it, and every consumer that skips it sees the raw data.

There is a second cost, which is duplication. If validation runs both at ingestion and again in the warehouse, the two implementations will diverge, and the divergence will be discovered when one rejects a record the other accepts. A single definition of the contract, expressed once and applied at one boundary, is the only arrangement that stays consistent as the schema evolves. The boundary itself is usually chosen by who can be forced to comply, not by where the check would be most effective.

Errors that pass every structural check

A record can satisfy its schema completely and still be wrong. A timestamp can be in the future. A count can be negative. A currency code can be a three-letter string that is not a currency. A customer identifier can be well-formed and belong to no customer. A total can fail to equal the sum of its parts.

These are the errors that justify a separate validation layer, and the layer’s checks are of a different kind: range predicates, membership in a reference set, arithmetic identities, and joins against a dimension. Each is a query, and each costs a scan. The arithmetic identities are the most valuable and the least used, because a check that a total equals the sum of its components catches errors that no per-column rule can, including errors introduced by the aggregation itself.

The division of labour is therefore not optional. Structural validation belongs at the boundary, where it is cheap, total, and enforceable. Semantic validation belongs after landing, where it is expensive, partial, and advisory, and where its failures are routed to a human rather than to a retry loop. A pipeline that treats the two as one layer will either enforce too much at the boundary and reject legitimate data, or enforce too little and accept data that is structurally perfect and factually false.