Snapshot Expiry, Reference Counting, and Files That Nothing Points To

Expiring a snapshot is a metadata operation. It removes an entry from the snapshot list in the table metadata document and commits a new version. The data files that the snapshot referenced are still on storage, unchanged, and the storage bill is unchanged with them. Nothing in the format’s commit path deletes a data file, because the commit protocol has no way to know whether some other snapshot, some in-flight reader, or some concurrent job still needs it.

The separation between dropping a reference and removing an object is the whole design. It is also where most operational mistakes in a lakehouse occur, in both directions: deleting objects that are still referenced, which corrupts old snapshots, and failing to delete objects that are not, which leaks storage indefinitely.

Reference counting over a set of snapshots, not one

The naive sweeper walks the current snapshot, collects the set of files it references, and deletes every other object under the table’s prefix. That is wrong in a way that is easy to miss and expensive to discover: the files belonging to retained-but-not-current snapshots are “every other object”, so the sweep destroys every historical version the table promised to keep, and destroys them silently. The next query against an older snapshot fails on a missing object, or worse, reads a partially swept snapshot if only some of its files were under the prefix.

The correct computation is over the union of all snapshots the table still retains:

$$R = \bigcup_{s \in \text{retained}} \text{files}(s) \cup \text{manifests}(s) \cup \text{deleteFiles}(s)$$

An object may be deleted only if it is outside $R$. Two details make this harder than the formula suggests.

First, the union must include the metadata documents themselves and the manifest lists, not just the data files. A manifest list referenced by a retained snapshot is needed to read that snapshot, and sweeping it produces an error even though every data file survives.

Second, delete files must be in the same union as data files. A position delete file is referenced by the snapshot that applied it; if the sweeper keeps the data file (because a later snapshot also references it) but deletes the delete file, the deleted rows reappear in that later snapshot. This is the resurrection bug described in delete files and merge-on-read, and it is caused by sweeping per-snapshot instead of over the union.

What expiry is allowed to remove

The retention policy has two independent knobs, and they are often confused.

Snapshot retention bounds how many historical versions remain readable. A policy might keep the last 10 snapshots plus any snapshot younger than seven days. After expiry, those versions are gone: time travel to them returns an error, and readers pinned to them fail.

File retention bounds how long an unreferenced object stays on storage before physical deletion. This is the window that protects in-flight readers, and it must be at least as long as the longest query. A reader that resolved snapshot $S$ at time $t$ and is still streaming at $t + 40$ minutes needs $S$‘s files to exist for 40 minutes; a sweeper that deletes unreferenced files after 30 minutes truncates that reader.

The two windows interact: a snapshot can be expired from the metadata (so no new reader can pin it) while its files remain on storage (so an existing reader survives). The gap between the windows is the safety margin, and setting them equal is the common misconfiguration.

There is a third, less obvious case: a failed commit’s leftovers. A writer that wrote data files, wrote a metadata object, and then lost the pointer swap leaves objects that no snapshot will ever reference. These are orphans from birth, and no retention policy over snapshots will ever collect them. The sweeper’s union-based rule handles them only if it compares against the union rather than against a snapshot list, and even then it must wait for the file-retention window because the writer may still be retrying.

How a sweeper enumerates candidates

The set $R$ is computable from metadata. The set of existing objects is not, on an object store: listing a prefix returns keys, and a large table’s prefix can hold millions of them. Listing is also eventually consistent in the sense that it can lag behind recent writes, so an object created seconds ago may not appear, which is harmless for deletion, since a missing candidate is simply not deleted.

The practical procedure is therefore a set difference:

1. Read the table metadata and enumerate every retained snapshot’s manifests, then every manifest’s data and delete files. This yields $R$.
2. List the table’s data prefix and the metadata prefix to get the candidate set $C$.
3. Delete $C \setminus R$, subject to an age filter that excludes anything modified within the retention window.

from datetime import datetime, timedelta, timezone

# `table` here is the store plus the format metadata; the calls below are
# the store's list and delete operations, not a library interface.
def sweep(table, now, file_retention):
    keep = table.referenced_paths()            # union over retained snapshots
    cutoff = now - file_retention
    deleted = 0
    for obj in table.list_prefixes():           # paginated listing
        if obj.path in keep:
            continue
        if obj.modified_at > cutoff:            # may belong to an in-flight writer
            continue
        table.delete_object(obj.path)
        deleted += 1
    return deleted

The age filter is doing real work. Without it, the sweeper races every writer: a writer creates a data file, and before it commits, the sweeper lists it, finds it unreferenced, and deletes it. The commit then succeeds and points at a file that no longer exists, producing a table whose current snapshot is unreadable. This race is not theoretical, and it is the reason that “run the sweeper after the writers finish” is not an acceptable control, since writers in a lakehouse rarely finish.

The two-sided cost of getting it wrong

Sweeping too aggressively produces errors that surface late and look like storage faults. A read of an old snapshot fails with a 404 on a manifest or data file; a scan of the current snapshot may fail if a concurrent rewrite’s inputs were removed before the rewrite’s outputs were committed. The recovery is a restore from a backup of the metadata, which most deployments do not have at the granularity required, so the practical recovery is to accept the loss of the affected versions.

Sweeping too conservatively produces a storage bill that grows without bound. A table that commits every two minutes and retains every snapshot accumulates 720 metadata documents per day, each a few hundred kilobytes, plus every manifest ever written: hundreds of megabytes a day of pure metadata, before any data. Since metadata documents reference their predecessors’ manifests, the retention policy must expire snapshots regularly or the manifest set grows monotonically even as the data is compacted.

There is a middle failure that is harder to diagnose: a sweeper that runs correctly but is never scheduled. Everything works, the table is correct, and the storage footprint grows at the rate of the commit log. Nothing alerts on it because no query is slow and no job fails. The symptom appears months later as a storage bill, and the cause is a missing cron entry rather than a bug.

Why the format cannot reclaim space on its own

A transactional database reclaims space because it owns the storage. It knows that a page is unreferenced because it maintains the free list itself, under the same lock as the allocation. A table format does not own the storage; it shares a prefix with other tables, other jobs, and whatever else an operator put there. It cannot know whether an object is garbage without enumerating, and it cannot enumerate without knowing the prefix boundaries, which is why every format’s maintenance tooling asks for a table location and refuses to operate outside it.

That boundary is a safety mechanism, not an inconvenience. A sweeper pointed at a bucket rather than a table prefix will delete objects belonging to other tables, and the union-based rule does not help because those tables’ references are not in the metadata it read. The blast radius of a mistyped prefix is every table under it.

The second structural limit is that reference counting requires reading every retained snapshot’s full manifest set. For a table with a thousand retained snapshots and hundreds of manifests each, the sweep reads hundreds of thousands of manifest entries before it deletes anything. That is why sweepers are batch jobs with their own resource budget rather than something that runs inside a query engine, and why the sweep’s own I/O can rival the table’s daily ingestion I/O on a table with a long retention window.

Retention as a correctness parameter

The retention window is usually set from a compliance requirement such as keep 30 days or keep 7 years, but it also determines what the table can guarantee. A window of zero makes time travel impossible and makes concurrent long queries unsafe. A window measured in years makes the metadata set large and the sweeper expensive. The setting that works is the one derived from the longest query and the longest job, plus the compliance requirement if it is longer, and the two numbers should be written down separately because they are adjusted by different people for different reasons.

One last interaction: a compaction job that rewrites a partition inside the retention window must keep its input files alive for the window, which transiently doubles that partition’s footprint. A retention policy that assumes single-copy storage will be surprised by the sweep’s timing, since the inputs cannot be collected until the window passes. The compaction job and the sweeper are therefore coupled through the retention setting, and changing one without the other changes the table’s storage profile.