Schema Evolution Without Rewriting Files

A table written in January has a column called customer_id. In March the column is renamed buyer_id and a new column is inserted before it. A reader opening the January file with the March schema must find the old column and bind it to the new name, and must not bind the new column to the old data. If the file identifies columns by position or by name, one of those two requirements fails. If it identifies them by a number that was assigned when the field was created and never reused, both hold.

Field identifiers are the mechanism that makes schema evolution a metadata operation instead of a rewrite. The design is easy to state and has several places where it is easy to get wrong.

Why names and positions both fail

Name-based binding fails on rename, which is the most common schema change in practice, and it fails silently in the other direction: a column dropped and a new column created with the same name will bind the new column to old data of a different type. Position-based binding fails on insertion, because every field after the insertion point shifts by one and every old file is misread from that point onward.

Both failures share a property that makes them dangerous: they produce plausible values rather than errors. A reader that binds buyer_id to an integer column of a different meaning returns integers, and the query returns a number. Only a type mismatch catches the mistake, and a rename between fields of the same type does not produce one.

Identifier-based binding moves the problem to the metadata. The writer records, per field, an integer that is stable for the life of the field. The reader resolves each field in the file against the reader’s schema by matching integers, and names become documentation rather than keys. Renaming is then free, reordering is free, and insertion is free.

The identifier’s lifecycle

The rules that make identifiers work are about reuse, and every format that uses them states a variant of the same three rules. An identifier, once assigned to a field, is never assigned to a different field. A field that is removed has its identifier retired, not returned to the pool. A field that is renamed keeps its identifier.

Protocol Buffers states this most explicitly, with field numbers from 1 to $2^{29}-1$, a reserved range for the implementation, and a reserved declaration that lets a schema mark numbers and names as permanently unusable so that a later editor cannot accidentally reintroduce them. The reserved declaration is the enforcement mechanism for the no-reuse rule, and it exists because the failure it prevents is invisible until an old file is read with a new schema.

Avro takes a different route to the same goal: fields are matched by name by default, and the aliases mechanism lets a schema declare that a new name is equivalent to an old one. Evolution is then a resolution step between writer schema and reader schema, with rules for what may change. The identifier is a name plus an alias set rather than an integer, which makes schemas more readable and makes the rename an explicit act rather than an implicit consequence of stability.

Parquet carries field identifiers in its schema tree as an optional integer per field, and the specification allows a reader to use them when present. Since the identifiers are optional and many writers omit them, a Parquet file is not self-describing with respect to evolution: two files with identical column names but different meanings are indistinguishable to a reader that only has names. This is why systems that need reliable evolution over Parquet maintain a separate schema registry and write the identifier into the file metadata.

The changes that stay illegal

Identifiers make renames, reordering, and insertion safe. They do not make every change safe, because some changes alter the physical interpretation of the stored bytes.

Widening an integer from 32 to 64 bits is safe in a format that stores integers in a width-tagged form and unsafe in one that stores them at the schema’s declared width. Narrowing is unsafe in both directions: an old value that exceeds the new type’s range has no representation. Changing a string column’s declared encoding or collation changes comparison order, which changes the meaning of every min/max statistic in the file. Changing a timestamp’s unit from milliseconds to microseconds multiplies every stored value by a thousand, and a reader that applies the new unit to old data produces timestamps in a different era. Changing a decimal’s scale shifts every value by a power of ten, the same failure in a different disguise.

Adding a required field is illegal against old data, because the old rows have no value for it and no default was ever written. The formats handle this differently: Avro resolution requires a default in the reader schema for a field absent from the writer schema, and refuses the read if none is given; Protocol Buffers gives every field a zero default, which is why a newly added field appears as zero rather than as absent in old messages; Parquet requires the reader to supply the value or to treat the field as null, and a required field with no value in an old file is a contradiction the reader must resolve by policy.

Removing a field is always safe for readers that ignore unknown fields, which is the standard forward-compatibility rule. It is not safe for readers that require every field in their schema to be present, so the reader’s behaviour on unknown and missing fields has to be stated by the system, not assumed.

Resolution as an explicit step

The cleanest formulation treats schema resolution as a distinct stage: read the writer’s schema from the file, read the reader’s schema from the query, produce a resolution plan that maps writer fields to reader fields and lists conversions, then decode with that plan. Avro’s specification describes exactly this, with rules for records, unions, enums, arrays, and maps, and a requirement that the resolution be performed before any data is read.

The plan is worth materializing because it is per-file and reusable across all row groups in that file, and because it separates two questions that are otherwise entangled: which fields to read, and how to convert them. A reader that resolves on the fly pays a branch per value; a reader with a plan can select a decoder per column and then run a tight loop.

# Resolve writer fields to reader fields by identifier, not by name.
def resolve(writer_fields, reader_fields):
    by_id = {f["id"]: f for f in writer_fields}
    plan = []
    for rf in reader_fields:
        wf = by_id.get(rf["id"])
        if wf is None:
            plan.append((rf["name"], "default", rf.get("default")))
        elif wf["type"] != rf["type"]:
            plan.append((rf["name"], "convert", (wf["type"], rf["type"])))
        else:
            plan.append((rf["name"], "copy", wf["name"]))
    return plan

The plan above deliberately drops writer fields that the reader does not ask for, which is the projection that makes columnar reads cheap; the identifiers are what let the reader skip them without guessing at positions.

Versioning the schema itself

Identifiers solve field-level evolution. They do not solve the problem of a schema that changes in ways the identifiers cannot express, such as splitting one field into two or merging two into one. Those changes require a new logical schema and a transformation, and the transformation is data, not metadata.

A related split runs through every file format that separates a physical schema from a logical one. Parquet’s schema tree records physical types, groups, and repetition, while the logical annotations record what those physical bytes mean: an INT32 annotated as a date, a BYTE_ARRAY annotated as a string, an INT64 annotated as a timestamp with a stated unit. The annotation is part of the contract, so changing it is a schema change even though the bytes are untouched, and the reader resolves it at the same stage as field binding. The three-level hierarchy that carries these annotations is described in Inside a Parquet File.

The cost of resolution is paid once per file, and the plan it produces is small: one entry per reader field, naming the writer field, the conversion, or the default. For a schema of a few hundred fields the plan is a few kilobytes and the binding work is a hash lookup per field, which is negligible against the cost of reading any page. The reason to make it an explicit stage is not performance but failure behaviour: a resolution step can reject an incompatible pair before any data is decoded, whereas a reader that resolves lazily discovers the incompatibility after it has already returned rows.

The usual structure is a registry that assigns each schema version an identifier and records the compatibility relationship between consecutive versions, so a reader can determine whether an old file can be read under a new schema before attempting it. The compatibility rules are the same ones described above, checked mechanically: a backward-compatible change is one where a new reader can read old data, forward-compatible is where an old reader can read new data, and full compatibility requires both. Adding an optional field with a default is backward-compatible; removing a required field is not.

The failure mode to watch for is a schema registry that validates compatibility but not identifier stability. A schema that renames a field and keeps the identifier passes every check; a schema that reuses a retired identifier for a new field also passes, and produces silently wrong reads. Identifier stability is a property of the editing process, not of any single schema document, which is why the reserved declaration in the format exists and why a review step is the only reliable enforcement. The same reasoning applies to column statistics in Column Statistics and the Page-Skipping Decision, where a type change retroactively invalidates every min/max recorded under the old type.