Small File Accumulation and What Compaction Must Preserve

A streaming job commits every two minutes. Each commit writes one file of a few hundred rows, because the job cannot buffer longer without delaying downstream readers. After a day the table has 720 files, each smaller than a single Parquet row group. Nothing about the table is wrong: every commit is atomic, every file is valid, every query returns the right answer. The table is simply slow, and it stays slow after the streaming job is retired, because the files are permanent.

This is the failure mode that compaction exists to repair, and the reason every mature lakehouse deployment has a maintenance job that nobody thinks about until it stops running.

Why many small files cost more than their bytes

The cost has four independent sources, and they compound.

Per-file overhead in planning. Every data file is an entry in a manifest, and every entry is parsed during planning. A table with 500,000 files has a manifest set of hundreds of megabytes even when the data is a few terabytes. Planning reads those bytes before it reads any data, so a query returning ten rows can spend most of its time in metadata.

Per-file overhead in execution. Each file opened costs at least one request to object storage, plus a footer read, plus a decompression of the footer’s metadata. Parquet footers carry column statistics and offsets, and their size grows with the number of columns and row groups. Opening ten thousand files to read a million rows spends more time in footers than in decoding values.

Lost compression. Parquet’s encodings, among them dictionary encoding, run-length encoding of definition levels, and delta encoding of integers, work better over longer runs of similar values. A file of 500 rows may have a dictionary as large as its data, and its row group may be too small for the encoder to amortise the dictionary’s header. A file of 500,000 rows compresses better per row, sometimes by a wide margin, and the gap is a property of encoding, not of the data.

Lost pruning. Statistics on a tiny file are nearly useless, because the file’s value range covers almost the whole domain. This is the ratio argument from manifest statistics and file pruning: a file of $n$ rows drawn from $D$ distinct values has a range covering roughly $n$ values, so small files prune poorly and many small files prune worse than one large one with the same total content.

The first two costs scale linearly with file count; the last two are superlinear in the sense that they get worse as files get smaller relative to the data. A table can be simultaneously small in bytes and pathological in file count.

What a rewrite must not break

Compaction is a transaction that removes a set of data files and adds a set of replacements. It runs concurrently with readers and writers, so the constraints are strict.

Readers on older snapshots must keep working. A reader pinned to snapshot $S$ holds paths to the files that $S$ references. If compaction’s commit removes those files from the current snapshot but $S$ is still retained, the files must remain on storage until $S$ expires. A compaction job that deletes its input files immediately after committing breaks every reader that started before it. The rule is that compaction only removes references; physical deletion is the job of the expiry process described in snapshot expiry and orphan files.

Delete files must be applied, not carried. If an input file has pending position deletes, the rewritten file cannot inherit them, because the positions no longer refer to the same rows. The rewrite must apply them. For equality deletes the rewrite may carry them forward, but the sequence numbers must be maintained correctly so that the deletes still apply to the rewritten rows and not to rows added later. This is the coupling described in delete files and merge-on-read.

Statistics must be recomputed, not copied. A replacement file’s bounds are the bounds of its actual contents. Copying the input files’ bounds produces a superset that is safe for pruning but useless; producing a subset by taking the intersection of the inputs’ bounds is unsafe and can drop rows. The only correct answer is to compute bounds during the rewrite, which means the rewrite must decode the values rather than concatenate bytes.

Sequence numbers and ordering must be preserved. Rows carry an implicit order through the file’s sequence number, and a rewrite that merges files from different commits must keep each row’s ordering relative to deletes and to later commits. Implementations that sort during a rewrite must preserve the row-level sequence number or record it explicitly, otherwise a rewrite can resurrect a row that a delete had removed.

The schema must be the one the file was written with. A rewrite reads files under possibly several schema versions. It must resolve each input’s columns by identifier and write output under one schema, which is the only point at which a rewrite can also perform a type promotion.

# Bin-packing inputs into output groups bounded by a target byte size.
# Sizes come from the manifest; no data is read at this stage.
TARGET = 512 * 1024 * 1024

def plan_groups(entries):
    groups, current, size = [], [], 0
    for e in sorted(entries, key=lambda x: (x.partition, x.size_bytes)):
        if current and size + e.size_bytes > TARGET:
            groups.append(current)
            current, size = [], 0
        current.append(e)
        size += e.size_bytes
    if current:
        groups.append(current)
    return [g for g in groups if len(g) > 1]  # singletons need no rewrite

The plan is deliberately metadata-only: grouping by partition and packing to a byte target reads no data, so a compaction run can be scoped and estimated before any I/O. A group of one file is dropped, because rewriting a single file at the same size achieves nothing except a new sequence number.

Thresholds that make the decision automatic

A maintenance job needs a rule for which files to rewrite. Three rules cover most cases.

Count-based: rewrite when a partition has more than $N$ files, or when the median file size falls below a target. This is cheap, uses only manifest data, and handles the streaming-commit case directly.

Deleted-fraction-based: rewrite a file when its deleted-row count exceeds some fraction of its row count. The counts are in the manifest, so this is also metadata-only. The threshold should be well under one, because a file at 30% deleted still reads 70% useful rows; the crossover is where the cost of reading and filtering exceeds the cost of rewriting, and it depends on the file size.

Age-based: rewrite files older than some interval. This has no principled basis and is mostly a proxy for the other two.

The interaction between count-based and deleted-fraction-based rules is worth noting: a compaction job that rewrites a partition to reduce file count will also apply deletes, so running it after a burst of deletes kills two problems with one rewrite. Running them as separate jobs rewrites the same bytes twice.

The rewrite’s own cost

Compaction is not free, and its cost is easy to underestimate because it reads and writes full files.

A rewrite of $B$ bytes reads $B$ and writes $B’ \le B$ (after applying deletes and recompressing). If the job runs while the table is being written, the total I/O for the table becomes the ingestion I/O plus the compaction I/O. On a table where ingestion is continuous and compaction runs daily over the whole table, the compaction I/O can exceed the ingestion I/O, which makes the maintenance job the dominant consumer of the storage tier’s throughput.

Two mitigations follow. Compaction can be scoped to recent partitions, since old partitions are already compact and do not change; this keeps the daily rewrite proportional to the day’s ingestion rather than to the table. And compaction can rewrite only the files that fail a threshold, which keeps a steady state where most files are large and only the tail is rewritten.

The steady state is the important idea. A table that compacts to a target file size, with ingestion producing files smaller than the target, reaches an equilibrium where the number of files is roughly the table size divided by the target size. If ingestion produces files at the target size to begin with, because the writer buffers to that size, compaction has nothing to do, and the cheapest maintenance job is the one the writer makes unnecessary. Buffer sizing at the writer is the highest-leverage fix, and it is a property of the ingestion job, not of the table.

When compaction is the wrong answer

When files are small but few. A table with a hundred 1 MiB files is not pathological; the overhead is a hundred footer reads. Compaction would rewrite 100 MiB to save a few milliseconds of planning.

When the table is append-only and partitioned by time. Old partitions never change, so a rewrite of them buys nothing except a one-time reduction in file count. If the files are already above the target, there is nothing to do, and a job that rewrites them anyway is pure waste.

When the read pattern does not prune. A table whose queries always scan everything gains nothing from better statistics, so the pruning benefit of compaction does not materialise. Only the per-file overhead benefit remains, and that is bounded by the file count.

When the rewrite would violate retention. A rewrite of a partition whose files are still referenced by snapshots inside the retention window must keep the inputs alive, which doubles the storage for the duration. On a table with a long retention window, the compaction job can transiently double the table’s footprint, and a storage budget that assumes single-copy must account for it.

The general principle: compaction converts write amplification now into read amplification later, and it is worth doing when the read side dominates. On a table read once a day, the trade is easy. On a table read continuously and written rarely, the files are probably already fine.