Data Engineering · Data Quality
Partition Statistics as Validation Signals
Per-partition row counts, byte sizes, and file counts are metadata, so checking them costs almost nothing; the hard part is knowing which deviation means which fault.
A partition that holds zero rows is the most informative state a table can be in, and it is also the one that most pipelines cannot distinguish from a partition that holds no files. Both look like nothing when a query scans them. Only the metadata separates them: one has a file list whose statistics sum to zero, the other has an empty file list.
That distinction is worth a section of its own because it is the difference between “the load ran and found no data” and “the load did not run”. The first is a fact about the source; the second is a fault in the pipeline. A validation layer built on partition metadata can tell them apart for the cost of reading a manifest.
The metadata that already exists
Every table format records, per file, the number of rows, the size in bytes, and per-column statistics including null counts and minimum and maximum values. It records, per partition, which files belong to it. It records, per snapshot, which partitions exist and which files were added or removed.
The cumulative result is that a partition’s row count is a sum over its file footers, and its byte size is a sum over file lengths. Neither requires opening a data page. Reading a Parquet footer costs a couple of seeks and a small read, on the order of kilobytes, regardless of whether the file holds a thousand rows or ten million. A manifest in a modern table format stores the same statistics inline, so an entire partition’s summary can be read from one file that is itself small.
This is the property that makes volume checks viable on every load rather than as a periodic audit. A check that costs milliseconds per partition can run on every commit, and one that costs a full scan cannot.
The metadata is also the reason volume checks have a known blind spot: it describes files, not values. A file can be rewritten with the same row count and different content, and the partition’s statistics will not change. Validation built on metadata detects changes in shape, not changes in meaning.
Four deviations and what each one means
A partition’s row count is compared against an expectation, and the expectation is what carries the information. Four deviations are worth distinguishing because they have different causes and different remedies.
A partition that is missing entirely means the load did not produce it. Either the scheduler did not run, or the job ran and wrote nothing because its upstream produced nothing. Distinguishing these needs a job-completion record; the table metadata alone cannot tell.
A partition that exists with zero rows means the job ran and wrote an empty file or no files. This is legitimate for a source with genuinely no events in that interval, and it is a fault when the source should have produced something. The signal that separates them is the row count of the input, not the output, which is why input-side counts belong in the same check.
A partition that is smaller than expected by a modest factor is the ambiguous case, and it is the one that absorbs most of the effort in any volume-check design. A ten percent shortfall can be a partial load, a source-side outage, a filter that excluded more rows than intended, or a legitimate drop in activity. Nothing in the partition metadata resolves the ambiguity; only a comparison against a second, independent expectation does, such as the source’s own count or the previous period’s count adjusted for trend.
A partition that is much larger than expected is usually a duplicate write or a join that fanned out. A partition that doubles in size overnight while its distinct key count stays constant is a duplicate write; one that doubles in size while the distinct count also rises is a fan-out or a genuine volume increase.
The four cases map onto a diagnostic that is more useful than the raw deviation: compare the row count and the distinct count of the partition’s key. The pair separates duplication from growth, which the row count alone cannot.
File count and the small-file signal
The number of files in a partition is a separate axis from the number of rows, and it moves independently.
A partition that holds the expected rows in ten times the expected number of files is not wrong in content, but it is wrong in shape, and the cost lands on every subsequent reader. Each file carries a footer to parse, a row group to plan around, and a unit of parallelism that may be far smaller than a task should be. A scan over a partition with a million small files spends its time on metadata rather than data, and the overhead is per file rather than per byte.
The signal is useful in both directions. A rising file count at constant row count indicates a streaming writer committing frequently without compaction, or a job whose output parallelism has increased. A falling file count at constant row count indicates compaction, which is healthy, and a falling file count at falling row count indicates a load that is overwriting rather than appending, which may be a bug in the write mode.
Average file size is the derived quantity worth tracking, and the comparison is against the block or row-group granularity of the format rather than an absolute ideal. A file significantly smaller than one row group cannot be read with the parallelism the format was designed for.
The expectation has to come from somewhere else
Every volume check needs a baseline, and the choice of baseline determines which faults the check can see. Three baselines are common, and each has a characteristic blind spot.
The previous comparable period is the simplest: compare today’s partition to yesterday’s, or this week’s to last week’s. It adapts automatically to growth and requires no maintenance. Its blind spot is the persistent change: a fault that lasts more than one period becomes the baseline, and the check reports a problem only on the first occurrence. It also fails at boundaries where the comparison is not like-for-like, such as a month with a different number of days or a period containing a holiday.
A moving average over several periods smooths the noise and tolerates weekly seasonality, at the cost of responding slowly. A step change of the size the check is meant to catch will be diluted across the window, so the deviation has to be larger to trigger. The window length is the trade: longer windows reduce false alerts and delay detection.
An expectation computed from the source is the most accurate and the most expensive. If the source system can report how many records it produced in the interval, that count is the ground truth against which the landed count should be compared, and the difference is the pipeline’s loss. The cost is a query against a system the pipeline may not control, and the availability of that query becomes a dependency of the check itself.
The three can be combined, and combining them is what makes a check hard to fool: the source count catches partial extraction, the period comparison catches scheduling faults, and the shape comparison catches fan-out. A check that uses only one will be silent for a whole class of faults.
Late and rewritten partitions break the comparison
Append-only assumptions fail in two ways that matter for partition checks.
Late data writes into a past partition. If the check compares each partition once, at the time it is first written, a later write that adds rows to it is invisible. If the check compares every partition on every run, then a partition that is legitimately still receiving late data will keep deviating from its baseline, and the check will alert repeatedly on correct behaviour. Neither arrangement is right; the check needs to know the partition’s expected finality, which is a property of the source’s lateness behaviour and not of the table.
Rewrites are the second problem. A compaction or an overwrite replaces a partition’s files, and the row count may change if the overwrite was an upsert that also removed rows. A check that treats any change in a closed partition as an anomaly will fire on every compaction. The remedy is to compare against the snapshot history rather than the current state: a change that comes with a commit whose operation is a rewrite is expected, and one that comes from an append is not. Modern table formats record the operation type per snapshot, which makes this distinction available from metadata.
The general form of both problems is that a partition is not a stable unit; it is a container whose contents change under a policy. Validation has to model the policy, or it will report the policy as a fault.
-- Partitions whose row count deviates from the trailing average of the previous week.
-- manifest_entries projects record_count, file_size_in_bytes and file_path per data file.
WITH per_partition AS (
SELECT partition_date,
sum(record_count) AS rows,
count(DISTINCT file_path) AS files,
sum(file_size_in_bytes) AS bytes
FROM manifest_entries
WHERE partition_date >= current_date - INTERVAL '14' DAY
GROUP BY partition_date
),
baseline AS (
SELECT partition_date, rows, files,
avg(rows) OVER (ORDER BY partition_date
ROWS BETWEEN 7 PRECEDING AND 1 PRECEDING) AS avg_rows
FROM per_partition
)
SELECT partition_date, rows, avg_rows,
rows::double / nullif(avg_rows, 0) AS ratio
FROM baseline
WHERE avg_rows IS NOT NULL
AND (rows < 0.5 * avg_rows OR rows > 2.0 * avg_rows);Counting is not the same as agreeing
A partition count that matches its baseline proves the pipeline moved approximately the right number of rows. It does not prove the rows are the right ones.
Two failures pass a count check with no deviation at all. A transform that drops a column and adds a constant produces the same number of rows with different content. A join that matches the wrong dimension rows, but the same number of them, produces the same count. In both cases the volume signal is exactly what it should be.
Detecting them requires content-level signals, which is what column profiles over null rates and cardinality provide, and what checksums over a canonical ordering provide for a stronger guarantee. A per-partition checksum computed as the hash of an ordered projection of the rows will differ for any content change, and it costs a scan of the partition. That cost is the reason checksums are usually applied to a sample or to a subset of columns rather than the whole table.
The layered arrangement is the one that works: metadata counts on every commit, because they are free; profiles on every commit for the columns that matter, because they are cheap; checksums and relational assertions on a slower schedule, because they are not. Each layer catches what the one above it cannot see, and the ordering by cost is also the ordering by frequency.
When a partition is the wrong unit
Partition-level checks assume the partition is the unit of failure, which is true when the writer commits per partition and false when it commits across many partitions at once.
A job that writes a hundred date partitions in a single commit can fail halfway and leave some partitions written and others empty, and the commit either succeeds for all of them or fails for all of them depending on the atomicity the format provides. Formats with atomic snapshot commits give all-or-nothing at the commit level, so a partial write is impossible after the commit, though a job that fails before committing leaves nothing at all. Formats or writers without atomic commits can leave a table with a subset of partitions populated, and then the per-partition check is the right unit and the table-level total is misleading.
The other direction is a partition whose rows come from several sources. A partition keyed by date and region, written by a job that processes regions independently, can have one region’s rows missing while the total is only slightly low. Checking the total will miss it; checking each partition will catch it. The partition key therefore has to be fine enough that a fault in any one writer affects one partition, and coarse enough that the number of partitions stays manageable. A key that is too coarse also defeats the freshness check, since a single partition cannot be partially fresh, and it is worth resolving the granularity once rather than twice.
References
- Apache Iceberg Specification — Table Metadata and Manifests — Apache Iceberg project
- Delta Transaction Log Protocol — Delta Lake project
- Apache Hive Language Manual — DDL — Apache Hive project
- Parquet File Format Specification — Apache Parquet project