Partition Evolution Without Rewriting Existing Data

Partitioning by day is the right choice until the table’s volume grows by two orders of magnitude and the daily partitions become files of a hundred thousand rows each, at which point partitioning by hour is the right choice. In a Hive-style table the migration is a rewrite: the partition key is encoded in the directory name, every file’s path contains the old key, and changing the granularity means reading every file and writing it under a new path. For a table of any size that is a full copy, and it must be done with the table offline or with two tables and a swap.

The lakehouse formats decouple the partition layout from the file path, which makes the migration a metadata change. The mechanism is unremarkable once stated, but it has consequences that reach into planning, statistics, and the meaning of a path.

The partition value moves into the manifest

In a Hive table the path is the partition: /sales/dt=2025-03-27/hour=14/part-00017.parquet names a file whose rows all have that date and hour. A reader derives the partition values by parsing the path, so the path must be consistent with the contents, and the format has no way to check that it is.

In the lakehouse formats, the partition values for each file are stored as a tuple in the manifest entry, alongside the file path. The path itself is opaque; nothing is inferred from it. A manifest entry carries a partition struct whose fields are ordered by the table’s partition specification, and the specification is itself a versioned object referenced from the metadata document described in the metadata tree.

Because the specification is referenced by identifier, the table can hold files produced under different specifications at the same time. Each manifest records which specification produced the files it describes, so a reader knows how to interpret the partition tuples it finds. Changing the partition layout means writing a new specification, incrementing its identifier, and using it for subsequent writes. Existing files keep their tuples and their spec identifier, and remain readable because the old specification is still in the metadata.

{
  "last-partition-id": 1001,
  "default-spec-id": 1,
  "partition-specs": [
    { "spec-id": 0, "fields": [
      { "source-id": 3, "field-id": 1000, "name": "order_day", "transform": "day" }
    ]},
    { "spec-id": 1, "fields": [
      { "source-id": 3, "field-id": 1000, "name": "order_day",  "transform": "day"  },
      { "source-id": 3, "field-id": 1001, "name": "order_hour", "transform": "hour" }
    ]}
  ]
}

The transform field is what makes the design more general than a column list. A transform is a function from a source column to a partition value: day, hour, month, year, bucket[N], or truncate[W]. Because the transform is applied by the writer and recorded in the spec, the source column does not need to appear in the table’s schema as a separate partitioning column. This is hidden partitioning: a query writes WHERE order_ts >= TIMESTAMP '2025-03-27 00:00:00' against the timestamp column, and the planner recognises that the predicate can be translated through the day transform into a partition filter, so files are pruned without the query mentioning a partition column at all.

Contrast that with Hive’s explicit partitioning, where order_day is a synthetic column and a query that filters on order_ts prunes nothing unless the writer also added a redundant predicate on order_day. The redundancy is a standing invitation to write a query that reads every partition, and it is the reason that “partition pruning” in Hive-era systems was so often ineffective.

What a query pays for a mixed layout

After an evolution, a table holds files under spec 0 and spec 1. A planner must handle both, and the cost is not zero.

Planning reads manifests grouped by spec, and the per-file bounds described in manifest statistics and file pruning are the fallback that keeps the coarse half of the table from being read in full. For spec 0, the partition tuple has one field and the predicate can only be narrowed to a day; for spec 1, the tuple has two fields and the same predicate narrows to an hour. The result is a union of file sets: the day-level files that survive the day filter, plus the hour-level files that survive the hour filter. Neither set can be pruned using the other’s granularity. A query for a single hour must read every spec-0 file for that whole day, because those files have no hour-level partition value.

This is the central trade of partition evolution: it is cheap in write bytes and expensive in read work until the old layout ages out. The mitigation is either to rewrite the old files, reintroducing the cost the feature was meant to avoid, or to let retention remove them, which only works for time-partitioned tables where old data expires. For a table partitioned by a non-temporal key, the old layout persists indefinitely, and a query that spans both eras pays the coarser granularity for the older half.

There is a second cost, in metadata width. Each partition specification adds a struct type, and manifests written under different specs are not byte-compatible, so a reader cannot merge them into one scan of a single manifest set. Implementations typically keep manifests segregated by spec, which means more manifest files for the same number of data files, and a slightly longer planning phase.

The constraint that makes evolution safe

The reason the old files stay valid is that the partition value is a function of the row, evaluated at write time. A file’s rows all map to the same partition tuple under the spec that produced it. That invariant is what allows the manifest to store one tuple per file rather than one per row. If a later evolution changed the transform such that a row’s partition value under the new spec differed from its value under the old one, the row would still be correctly described by its own file’s tuple, and the old tuple remains a true statement about the old file.

What breaks the invariant is a write that produces a file whose rows span multiple partition values under its own spec. That is a writer bug, not a format limitation, and it produces a table where pruning silently returns wrong answers. Some writers avoid it by construction (a shuffle by partition key before write); others check it. Nothing in the format enforces it, which is the same class of hazard as a Hive writer putting rows in the wrong directory, except that in Hive the mistake is visible in the path and here it is not.

Transform choice as a physical decision

The transforms are not interchangeable, and their costs differ in ways that matter more than the partition count.

day and hour produce partition values with a natural ordering, so range predicates translate directly into contiguous partition ranges. bucket[N] hashes the source value modulo $N$, which destroys ordering: a predicate on the source column cannot prune buckets at all, and the transform exists purely to spread writes and to enable partition-wise joins on the bucketed key. truncate[W] keeps ordering, because it truncates a value to its first $W$ characters or to a multiple of $W$ for numerics, so a range predicate maps to a range of truncated values.

The practical consequence: bucketing is a layout decision for joins and write distribution, not for filtering. A table bucketed by customer_id and queried by order_date has no useful partition pruning at all, and relies entirely on file-level statistics. Mixing a temporal partition field with a bucketed one is possible under a multi-field spec, but each field multiplies the partition count, and a spec with a day and a 256-bucket field produces 256 partitions per day.

Choosing when to evolve

The signal to evolve is not “the partitions are too big” but “queries are reading too many files”. Those differ because partition size is a property of the writer and pruning is a property of the predicate. A table with one giant partition per day and an equality predicate on a high-cardinality column may prune perfectly well via file statistics if the data is clustered; a table with small hourly partitions and a predicate on an unpartitioned column prunes nothing.

Evolution is worth its read cost when the new granularity eliminates a large fraction of files for the dominant query shape and the old layout’s files are a small or shrinking share of the table. When the old files dominate, a rewrite is cheaper than living with the mixture, and a rewrite is exactly what the metadata-only change was designed to avoid. The feature removes the mandatory cost, not the eventual one.