Catalogs, Table Identity, and What a Table Name Resolves To

A query names sales.orders. Between that name and the bytes on storage there are at least two indirections: the name resolves to a table location, and the location resolves to a metadata document, and the metadata document resolves to a snapshot. Each indirection exists to allow something to change without breaking the others, and each is a place where a deployment can be configured inconsistently.

The catalog is the component that owns the first indirection. It is easy to treat as plumbing, a service that stores strings, but its behaviour determines whether a commit is atomic, whether two engines see the same table, and whether a rename is a metadata operation or a copy.

Two resolutions, not one

The name-to-location step is a namespace lookup: a catalog maps a multipart name (catalog, database or namespace, table) to a table location and, in the modern design, to the current metadata document’s path or identifier. The location-to-snapshot step is the table format’s own resolution, reading the metadata document and selecting a snapshot.

Splitting them this way is what allows a table to move between storage locations without its name changing, and what allows two tables with different formats to live in one namespace. It also means the two steps can disagree. A catalog entry pointing at a location where no metadata document exists is the most common form of a broken table: the name resolves, the location is a valid prefix, and the metadata fetch returns nothing. The error surfaces as “table not found” from an engine that has already resolved the name, which is why diagnosis usually requires checking the catalog and the storage separately.

Delta Lake’s original design collapses the first indirection: the table is the directory containing _delta_log, and a name is resolved to that directory by whatever external mechanism the engine uses. That is simpler and works with any storage, at the cost of making rename and location change operations that copy the log. Iceberg’s design keeps the indirection explicit, so a catalog can repoint a table at a new metadata location and a rename becomes a change to one catalog row.

The pointer swap

The commit protocol described in atomic commits and snapshot isolation needs a serialization point. The catalog provides it: the writer computes a new metadata document, writes it as an immutable object, and then asks the catalog to replace the table’s current metadata pointer, conditional on the pointer still being the value the writer read. If the condition holds, the commit is visible; if not, the writer lost the race and retries.

This is why a catalog is more than a name registry. Its compare-and-swap on the pointer is the operation that makes concurrent commits safe, and its availability is the table’s write availability. An object-store-only deployment has to substitute a weaker primitive, create-if-absent on a versioned metadata file name, and then “which metadata is current” becomes a question answered by listing and sorting, with the concurrency guarantee reduced accordingly.

The pointer also carries a version identifier, and readers use it for caching. A reader that caches a parsed metadata document keyed on the pointer’s version invalidates the cache when the pointer changes, which is a single small read rather than a re-parse of a large document. Without a version in the pointer, cache invalidation requires comparing timestamps or re-reading the whole document, and clock skew between writers makes the timestamp comparison unreliable.

-- Illustrative: the catalog stores one row per table with the current
-- metadata location and a version. Session A reads the pointer at
-- version 41 and starts writing.
SELECT current_version FROM catalog.tables WHERE name = 'sales.orders';
-- 41

-- Session B commits version 42 while A is still working.
-- A's conditional pointer update now fails: the pointer is no longer 41.
UPDATE catalog.tables SET metadata_location = :new_location, current_version = 42
 WHERE name = 'sales.orders' AND current_version = 41;
-- 0 rows updated -> A re-reads and retries

Namespaces, and what they do not guarantee

A catalog organises tables into namespaces, usually two levels: a database and a table name, sometimes three with a catalog prefix. The namespace is a naming convenience with an important non-guarantee: it does not imply a storage location, and it does not imply isolation.

Two tables in the same namespace can live in different buckets. Two tables in different namespaces can share a prefix, which is a configuration that a sweeper will eventually corrupt, because the sweeper’s union-based rule from snapshot expiry and orphan files assumes the prefix belongs to one table. Co-locating multiple tables under one prefix is legal on storage and unsafe in practice; the format has no way to distinguish another table’s files from its own orphans.

Namespaces also carry no concurrency semantics. Creating a table in a namespace is not atomic with respect to creating another table in the same namespace unless the catalog enforces it, and the enforcement is per-catalog rather than per-format. Two engines pointed at the same storage prefix but at different catalogs can both create “the same” table and neither will notice, producing two tables with one name that diverge from the moment of creation.

Identity, location, and the drift between them

The table’s identity is its universally unique identifier, recorded in the metadata document, not its name and not its path. The identifier is generated at creation and never changes. This is what allows a table to be renamed, moved, or restored from a copy without becoming a different table.

The identifier also provides a check that a catalog entry points at the right thing. A reader that resolves a name to a location, then reads a metadata document whose identifier does not match the one the catalog recorded, has found a mismatch: a table that was overwritten, a catalog repointed incorrectly, or a copy that was made by cloning files rather than by a table operation. Implementations that compare identifiers catch this; implementations that trust the path read the wrong table’s data and return it as if it were the right one, which is the worst possible failure because it produces plausible answers.

Cloning makes the distinction concrete. A shallow clone copies the metadata and reuses the data files, giving a new table with a new identifier and the same underlying files. That is safe because the data files are immutable and both tables only reference them. A deep clone copies both. A copy performed at the storage level, such as aws s3 cp --recursive, copies the files but produces a table whose metadata still records the original identifier, so two tables now claim the same identity, and any tool that keys on identity will treat them as one.

What the catalog must expose to readers

Beyond name resolution, a reader needs three things from the catalog, and the absence of any of them shows up as a performance or correctness problem rather than as a missing feature.

The current pointer, cheaply. A per-query round trip to the catalog is acceptable; a per-file round trip is not. Engines cache the pointer for the duration of a query and re-check it between queries.

Commit coordination. The conditional update, and a way for a writer to learn that it lost. Without it, writers must implement their own coordination on storage, which is where table formats without catalog support end up.

A listing of tables for discovery. Planning across many tables, such as a job that reads a hundred partitions of a hundred tables, needs to resolve names in bulk. A catalog that answers one name per request makes such a job’s startup time proportional to the number of tables, which is why implementations cache the namespace listing and why catalogs expose bulk operations.

None of these require the catalog to store data or statistics. A catalog that grows into a query planner is a different system, and the formats deliberately keep the catalog’s contract small so that any service can implement it.

Failure modes specific to the catalog layer

Split brain between two catalogs. Two catalog services over the same storage prefix, each with its own view of the pointer, produce two writers that both believe they committed. The table then has two divergent metadata documents and no way to reconcile them, because the format’s conflict detection runs inside one catalog. The only safe configuration is one authoritative pointer per table.

The pointer and the metadata disagree. A commit that wrote the metadata object but failed before the pointer update leaves an unreferenced metadata file, which is harmless. The reverse case, a pointer updated to a location that does not exist, happens when the metadata write is acknowledged but not durable, which some object stores permit for a window. Committing the pointer only after a successful read-back of the metadata object closes this, at the cost of one extra round trip per commit.

Renames that are not atomic with in-flight readers. A rename changes the name-to-location mapping. A reader that resolved the old name and is mid-query continues to read the old location, which is correct as long as the old location still exists. If the rename also moved the files, the reader fails. Formats handle this by making rename a catalog operation that changes only the mapping, leaving the files where they are.

Credential and permission drift. The catalog may grant a reader access to the table while the storage policy denies access to the prefix, or the reverse. Because the two systems have separate access control, a table can be listable and unreadable. Diagnosing it requires testing the two layers independently, and the error message from an engine usually names only one of them.