Atomic Commits and Snapshot Isolation Between Concurrent Writers

Two writers append to the same table at the same instant. Neither can see the other’s uncommitted work, and neither is willing to block. After both finish, the table must contain both appends or neither, and a reader must never observe a state that includes half of one. On a filesystem this is nearly free: rename is atomic, and a lock is available. On an object store there is no rename and no lock, and the only primitive is a conditional single-object write. Every table format is, at bottom, a scheme for building a serializable commit protocol out of that one primitive.

The commit as a compare-and-swap on one name

The metadata document is the whole state of the table at a version. Publishing a version means installing a new document under the name that readers resolve. If the store offers “create this object only if it does not exist”, or “replace this object only if its current version matches the one I read”, the commit is a single compare-and-swap and the race is resolved by the store.

Two families of implementation follow from which primitive the store actually offers.

Delta Lake uses the first. Its log is a directory of numbered JSON files, one per commit, named with a zero-padded 20-digit version: 00000000000000000000.json for the first commit, 00000000000000000001.json for the second, and so on. A writer that has read up to version $v$ attempts to create the file for version $v+1$ under a mutual-exclusion guarantee. Exactly one writer succeeds; the loser sees that the version is taken and retries from the new one. Where the object store cannot express create-if-absent, the guarantee comes from an auxiliary coordination service rather than from the store itself, which is why a Delta table on such a store is not writable by a client that has only storage credentials. Readers reconstruct state by reading the log in version order, so the file naming is the serialization point and the sequence of names is the commit order. Later versions of the protocol add checkpoint files in Parquet so that reconstructing state does not require replaying every JSON file since the table was created.

Iceberg uses the second. Its metadata document has a version number in the file name (v41.metadata.json, v42.metadata.json), and the commit is a swap of the pointer that names the current one, performed by a catalog that supports compare-and-swap on that pointer. Where the catalog cannot do this, as with a plain object-store prefix and no catalog service, the commit degenerates to create-if-absent on the next version name, which serializes writers but makes the “which file is current” question a matter of scanning for the highest number.

The pointer swap is what makes the format’s guarantee precise: readers resolve the pointer once and then read immutable objects reachable from it. A reader that resolved v41 never sees v42’s effects, no matter how long it runs. That is snapshot isolation, and it holds for reads without any read lock.

# Sketch of the optimistic commit loop; the store's conditional write is the CAS.
MAX_RETRIES = 4

def commit(table, read_version, produce_metadata):
    for attempt in range(MAX_RETRIES):
        base = table.read_metadata(read_version)
        candidate = produce_metadata(base)          # may read data, compute stats
        target = table.next_version_name(read_version)

        if not table.create_if_absent(target, candidate):
            read_version = table.current_version()   # someone else committed
            continue                                 # re-derive from the new base

        if table.compare_and_swap_pointer(read_version, target):
            return target
        table.delete(target)                         # lost the pointer swap
    raise CommitConflict("gave up after retries")

The deletion in the failure path matters. A commit that wrote its metadata object and then lost the pointer swap leaves an unreferenced object behind, and nothing will ever clean it up unless the failure path does so or a separate sweep handles it. This is the origin of a large fraction of the orphan files discussed in snapshot expiry and orphan files.

What the retry can and cannot redo

Optimistic concurrency is only viable if the work before the conflict check can be repeated cheaply. For an append, it can: the writer has already produced its data files, those files are immutable and unreferenced by any snapshot, and the retry only needs to rebuild a manifest entry against the new base. The data write is done once; the metadata derivation is repeated.

For a rewrite, it cannot be repeated cheaply, and this is where formats differ most in practice. A compaction job that rewrote a thousand files and then discovers a conflicting commit must decide whether to redo the rewrite or to rebase its output onto the new base. Rebasing is possible when the conflict is disjoint, meaning the other writer added files in partitions this job never touched, because the job’s new files still represent a valid replacement for the specific files it read. It is not possible when the other writer modified one of the files this job read, since the replacement no longer covers the current contents.

Formats express this as a conflict-detection rule over the set of files the transaction read and wrote. The check at commit time is set intersection: if any file this transaction removed was also added or removed by a commit that landed after this transaction started, the transaction conflicts. Append-only transactions remove nothing, so two concurrent appends never conflict. A transaction that replaces files conflicts with any transaction touching those same files, and with nothing else. This is a coarse but sound rule, and it is why partition-scoped compaction jobs can run concurrently with appends as long as they stay in their own partitions.

The failure mode worth naming is the false conflict. Two compaction jobs each rewrite a different subset of the same 500 small files in the same partition will not conflict if their file sets are disjoint, but two jobs that both read the same 500 files and each produce a different consolidated file will conflict even though either outcome is correct. Retrying the loser doubles the rewrite cost. Scheduling maintenance to avoid overlapping the same file set is cheaper than making the protocol smarter.

Serialization versus isolation

It is worth separating two guarantees that are often conflated. Serialization is about the order in which commits become visible; isolation is about what a reader is allowed to see. The commit protocol gives serialization for free: commits are ordered by version number, and that order is total. Isolation is the property that a reader at version $v$ sees exactly the state at $v$.

Snapshot isolation is weaker than serializability, and the gap is a real anomaly: write skew. Two transactions each read the table, each decide independently that some invariant holds, and each write a change that individually preserves the invariant and jointly breaks it. Because neither wrote a file the other read, the set-intersection conflict rule does not fire, and both commit. A concrete case is a table where a job enforces “at most one row per customer per day” by reading the day’s partition, finding none for customer 42, and appending one. Two such jobs racing both find none and both append, producing two rows. The protocol cannot prevent this because the write set (each job’s new file) is disjoint from the read set that mattered (the absence of a row).

The mitigations are all outside the format. A catalog with a lock, or a serializable isolation level at the catalog, closes the gap at the cost of throughput. A table property that routes all writes for a partition through one writer closes it by construction. Deduplicating at read time with a merge-on-read delete file closes it for readers that apply deletes, which is why delete files and merge-on-read is often adopted for exactly the tables where write skew matters.

Where the protocol is weakest

Clock-based conflict detection. Some implementations timestamp commits and use timestamps to order them, which imports clock skew into the commit order. Two writers whose clocks differ by seconds can produce a commit order that disagrees with the version order, and a reader using timestamps for time travel can miss a commit. Version numbers avoid this entirely, and any format that lets a timestamp stand in for a version number inherits the problem.

Catalog availability as a single point of failure. When the pointer swap lives in a catalog service, that service’s availability is the table’s write availability. Object-store-only commits trade this for weaker guarantees about which metadata file is current.

Retry livelock. A hot table with many small concurrent appends can produce enough conflict-and-retry cycles that writers make little progress. The protocol’s answer is batching, meaning fewer and larger commits, which is a design instruction to the writer, not a property of the format. The relationship between commit frequency and file count is the same one that drives compaction.

Readers and expiry. Snapshot isolation protects a reader from concurrent writes, but not from a maintenance job that removes the snapshot the reader pinned. The isolation guarantee is bounded by the retention window, and a reader that outlives the window gets errors, not a consistent older view.