Why Columns Need Identifiers Rather Than Names

A Parquet file written a year ago stores a column called customer at position 4 of its schema. Today the table calls that column customer_id and it sits at position 1, because three columns were dropped and one was inserted. A reader that resolves columns by name finds no customer_id in the old file and returns nulls. A reader that resolves columns by position reads the wrong column entirely and returns values from whatever now occupies position 4. Both are wrong, and both are the natural consequence of treating a name or an offset as the identity of a column.

Lakehouse table formats answer this by assigning each column an integer identifier at creation and never reusing it. The identifier, not the name, is the key that joins a query’s schema to the schema embedded in an old data file.

Identity as a monotonic integer

Every field in the table schema carries a field identifier. The identifiers are assigned when the field is added, they are unique within the table, and they are not reused even after the field is dropped. The schema itself is a tree, and identifiers apply to nested fields as well as top-level ones, so a struct’s member and a list’s element each have their own.

The metadata document stores the schema as a list of versions, each with a schema identifier. A reader at snapshot $S$ uses the schema that was current when $S$ was committed, not the latest schema. That distinction is what makes time travel correct rather than merely approximately correct: reading version 41 returns the columns as they existed at version 41, including columns that have since been dropped.

{
  "schemas": [
    { "schema-id": 0, "type": "struct", "fields": [
      { "id": 1, "name": "order_id",    "required": true,  "type": "long" },
      { "id": 2, "name": "customer",    "required": false, "type": "string" },
      { "id": 3, "name": "amount",      "required": false, "type": "decimal(18,2)" }
    ]},
    { "schema-id": 1, "type": "struct", "fields": [
      { "id": 1, "name": "order_id",    "required": true,  "type": "long" },
      { "id": 2, "name": "customer_id", "required": false, "type": "string" },
      { "id": 3, "name": "amount",      "required": false, "type": "decimal(18,2)" },
      { "id": 4, "name": "channel",     "required": false, "type": "string" }
    ]},
    { "schema-id": 2, "type": "struct", "fields": [
      { "id": 1, "name": "order_id",    "required": true,  "type": "long" },
      { "id": 2, "name": "customer_id", "required": false, "type": "string" },
      { "id": 4, "name": "channel",     "required": false, "type": "string" }
    ]}
  ],
  "current-schema-id": 2
}

Renaming customer to customer_id between schema 0 and schema 1 changes only the name; identifier 2 is untouched, so the old files still map correctly. Dropping amount between schema 1 and schema 2 removes the field from the current schema but leaves identifier 3 reserved. A later attempt to add a column named amount gets a fresh identifier, which is what prevents a new column from being silently filled with an old column’s data.

Delta Lake reaches the same guarantee by a different route: its log records the schema, and column mapping can be enabled so that Parquet files store physical column names that are stable identifiers while the logical names remain free to change. The mechanism differs, the requirement is identical: some stable key must survive a rename.

What the reader does with the identifier

A scan resolves each projected column to an identifier, then asks each data file’s footer for the column whose embedded field identifier matches. Parquet files written by a format-aware writer store the identifier in the field metadata, so the lookup is exact. Files written by a writer that did not store identifiers, such as an older engine or a plain Parquet dump, fall back to name matching, which is correct only if the name has never changed since that file was written. This is the practical boundary of schema evolution: the guarantee holds for files the format wrote, and is best-effort for files it merely adopted.

Two failure modes follow directly. If an adopted file’s column names do not match the table’s current names, the reader cannot map them and either fails or returns nulls for that file. And if two adopted files used the same name for different content, the table cannot tell them apart, because there is no identifier to compare. Both are reasons to import existing data through a rewrite rather than registering files in place when schema history is uncertain.

Type promotion, and the promotions that are refused

A schema change is legal in two forms: adding or dropping a field, and widening a field’s type. Widening must preserve every existing value, so the permitted set is small and directional.

From	To	Safe because
int	long	every 32-bit value is a 64-bit value
float	double	the widening is exact for the value set
decimal(P,S)	decimal(P',S) with $P’ > P$	more integer digits, same scale

That list is the floor rather than the ceiling: later format versions have added promotions to it, and the authoritative set for a given table is the one its format version defines. The reverse directions are refused, and the refusal is not pedantry. Narrowing long to int fails for any value outside the 32-bit range, and a reader cannot know in advance whether such a value exists without reading every file. Decimal scale changes are worse: decreasing the scale requires rounding, which changes values, and increasing it changes the physical representation in some encodings. Adding a required column needs a default value, because old files have no value for it and a reader must be able to synthesise one; a required column with no default would make every existing file invalid. Format versions differ in whether they support default values at all, and a version without them forces such a column to be optional with the null semantics that implies.

Note the asymmetry in the failure: a widening that a reader does not implement produces a decode error, not a wrong answer. A promotion applied to a file that was not rewritten produces values that must be reinterpreted at read time, an int stored as 32 bits read as 64, which requires the reader to know the file’s type as well as the table’s. That is why the type lives in the file footer and why the format’s rule is that the reader promotes from the file’s type to the table’s type, never the reverse.

Why name-based resolution cannot be repaired

The tempting fix for name-based schemas is to keep a rename history. This works until the history has a gap (a table restored from a backup, a file produced by a tool that did not record history, two branches of history merged), and then a rename is indistinguishable from a drop-and-add. The identifier scheme avoids the problem by construction: identity is assigned once, at creation, and never derived from anything that can change.

The cost is that identifiers must be allocated in a way that is safe under concurrency. Two writers adding a column simultaneously must not assign the same identifier to both. Implementations solve this the same way they solve any concurrent metadata change: the commit that adds the column fails its conflict check if another commit added one first, and the retry re-reads the schema and allocates the next identifier. A column addition is therefore a metadata commit with the same serialization properties as any other, which is why atomic commits is a prerequisite for schema evolution rather than an unrelated topic.

The interaction with pruning

Field identifiers have a second, less obvious role in the manifest statistics described in manifest statistics and file pruning. The bounds map in a manifest entry is keyed by field identifier, not by name. After a rename, the bounds recorded for identifier 2 still apply to the column now called customer_id, so pruning keeps working across the rename with no metadata rewrite. A name-keyed statistics map would lose its bounds at the rename and start reading every file.

The same keying explains why the bounds map covers a bounded set of columns: the format takes the first 32 fields in schema order, so dropping a field from the middle of the schema does shift which columns are covered, and a table whose hot predicate column sits past that limit gets no manifest-level pruning for it.

Where it still hurts

Renamed partition fields. A partition specification names its fields and references source columns by identifier, so a rename is safe for partitioning too. But the partition field names appear in the manifest tuples, and a rename of the partition field itself is a new specification, which puts the table into the mixed-layout state described in partition evolution.

Dropped columns leave data behind. Dropping a field removes it from the schema but does not rewrite existing files, so the bytes remain until those files are rewritten or expired. A dropped column that held sensitive data is still readable from an old snapshot by anyone with the snapshot identifier. Deletion of the data is a rewrite, not a metadata change, and treating a drop as an erasure is a compliance error.

Nested evolution is partial. Adding a field inside an existing struct works because the struct’s other members are unaffected. Renaming a struct member works for identifier resolution. But changing a struct’s shape, such as turning a struct into a list or reordering map keys, is not a promotion at all, and the format requires a drop-and-add with a rewrite.