How a Table Format Replaces Directory Listing with a Metadata Tree

A directory of Parquet files under a common prefix is not a table. It is a pile of files that happens to share a path. Every question worth asking about a table has no answer in a listing: which rows are in it right now, which files belong to version 41, whether two files written five minutes apart belong to the same consistent state. The listing returns names and sizes. It does not return membership, and it cannot return membership atomically.

The lakehouse table format exists to answer those questions. Its central move is to stop treating the prefix as the table and instead maintain an explicit, versioned description of which files constitute the table at a point in time. This article walks that description from the top down, because the shape of the chain explains nearly every cost and failure mode the format has.

Three levels, and why the middle one exists

The chain has a fixed spine in every open table format, even where the file names differ:

1. A table metadata document, usually JSON, naming the current version and carrying the schema, the partition specification, table properties, and a list of snapshots.
2. A manifest list, one per snapshot, listing the manifest files that belong to that snapshot.
3. Manifest files, each listing data files with per-file statistics and the partition values those files carry.
4. The data files themselves, typically Parquet or ORC.

The obvious objection is that levels two and three are redundant: the metadata document already knows the snapshot, so the manifest list appears to add a hop with nothing in it. The reason it exists is write amplification. A table with a million data files would need a million entries somewhere. If those entries lived directly in the metadata document, every commit, even one that adds a single file, would rewrite the entire document. With the extra level, a commit writes one new manifest holding the new files, one new manifest list that references the previous manifests plus the new one, and one new metadata document that points at the new snapshot. The bytes written per commit depend on the number of manifests, not on the number of data files.

Implementations roll a manifest at a byte target of a few megabytes rather than at a fixed entry count, which keeps a manifest small enough to read in one object-store request while amortising per-manifest overhead across thousands of file entries. A table with a million files therefore has on the order of a few hundred manifests, and the manifest list that names them is on the order of tens of kilobytes.

{
  "format-version": 2,
  "table-uuid": "9f1c4a2e-6b7d-4c31-8a05-2d1e77b3c9f4",
  "location": "s3://warehouse/sales",
  "current-snapshot-id": 3051729675574597004,
  "snapshots": [
    {
      "snapshot-id": 3051729675574597004,
      "timestamp-ms": 1743033600000,
      "sequence-number": 41,
      "manifest-list": "s3://warehouse/sales/metadata/snap-3051729675574597004.avro",
      "summary": { "added-data-files": "12", "total-data-files": "1841" }
    }
  ],
  "snapshot-log": [
    { "timestamp-ms": 1743033600000, "snapshot-id": 3051729675574597004 }
  ]
}

The snapshot-log array is the append-only history of which snapshot was current at which time. It is what makes time travel by timestamp possible: find the last entry whose timestamp precedes the requested instant, and read that snapshot. The array grows with the number of commits, so the metadata document is rewritten on every commit and grows slowly over the life of the table, one of the few places where cost scales with commit count rather than with data volume.

What a reader actually opens

Opening a table is a sequence of narrowing reads, and each step exists to avoid the next one.

The reader fetches the metadata document. In a catalog-backed setup this is one small object, though the location may itself be resolved through a catalog pointer, a subject treated in catalogs and table identity. It picks the current snapshot or the snapshot matching a requested version. It reads that snapshot’s manifest list, which is a single Avro file of a few hundred records. It can then discard most manifests without reading them, because each entry in the manifest list carries the partition field summaries and a count of files, and the query’s partition filter eliminates manifests whose summary ranges cannot overlap the predicate. Only for surviving manifests does it read the file entries, and only then does it produce a list of data files to hand to the scan.

The point of the three-level structure is that planning touches bytes proportional to the number of surviving files, not to the total. A query restricted to one day of a table partitioned by day reads one metadata document, one manifest list, and the handful of manifests whose partition summaries include that day. The other manifests are named but never opened. The statistics that make this possible in the file-level case are covered in manifest statistics and file pruning.

Contrast this with the pre-format convention of encoding partition values in the path, as in .../dt=2025-03-27/part-00017.parquet. A reader that wants one day lists that one prefix, which is cheap. A reader that wants the whole table lists every partition prefix, and the number of requests scales with the number of partitions, since a single list call returns at most 1000 keys on S3. Worse, the set of partitions is discovered by enumeration, so two writers adding partitions concurrently produce a state that was never committed as a unit. There is no version number, and there is no way to ask what the table looked like an hour ago.

The commit is a pointer, not a mutation

Because the metadata document is immutable and a new one is written for each commit, publishing a version reduces to making the new document the one that readers resolve. The mechanism for that swap, and the concurrency rules around it, are the subject of atomic commits and snapshot isolation; the structural consequence belongs here.

The consequence is that a snapshot is a reachability set. A data file belongs to snapshot $S$ if some manifest in $S$‘s manifest list names it. Nothing about the file’s location, name, or modification time determines membership. Two files written seconds apart can belong to different snapshots; a file written an hour ago can belong to both. This is what allows a table to hold both an old and a new version of the same logical row without a rename or an overwrite, and it is why deleting data is usually a metadata operation, a new manifest omitting the file, rather than a delete against storage.

It also means the format can be implemented over storage that offers no rename and no atomic multi-object operation. Object stores give per-object atomicity for a single PUT and, in current implementations, strong read-after-write consistency for new objects and overwrites. That is enough: if the commit is expressed as “write this new metadata document, then make it the one that counts”, the first half is a single-object write and the second half is whatever the catalog provides.

Where the design leaks

Three costs are structural rather than incidental.

Planning is a chain of dependent reads. Each level cannot be fetched until the one above it is parsed, so opening a table costs at least four serial round trips before any data is read: metadata, manifest list, manifest, data file. On object storage where a round trip is a network exchange measured in tens of milliseconds, that latency floor is visible on small queries. Formats mitigate it with caching of the metadata document and manifest list at the client, and by parallelising manifest reads once the list is known, but the first read of the day pays the chain in full.

Metadata grows with commits. The snapshot list and snapshot log in the metadata document are append-only, and each commit rewrites the document that contains them. Long-lived tables with frequent small commits accumulate thousands of snapshot entries. Expiring old snapshots bounds this, at the price of losing the ability to read those versions, a trade examined in snapshot expiry and orphan files.

Readers pin state that writers may delete. A reader holds a snapshot identifier and a set of manifest paths. Nothing prevents a concurrent maintenance job from expiring that snapshot and removing its manifests and data files, at which point the reader’s next fetch fails with a missing-object error rather than a stale answer. The format’s answer is a retention window and a rule that expiry must not remove files that any live snapshot references; enforcing it against readers that started before the sweep is the hard part, and it is the reason long-running scans and aggressive expiry interact badly.

The tree also has a degenerate case worth knowing: a table with zero commits has no current snapshot, and a reader must treat that as an empty table rather than as an error. Implementations that assume current-snapshot-id is always present fail on freshly created tables, which is the most common way a first query against a new table returns a null-pointer error instead of zero rows.