Data Validation at Pipeline Boundaries

A pipeline that moves bytes reliably can still deliver the wrong numbers. The transport promises delivery; it does not promise that the column called price is still in the currency the model was trained on, that the join still matches the same rows, or that today’s partition is not half empty because an upstream job failed at 03:00 and succeeded at 09:00. Validation is the set of assertions a pipeline makes about its data at the points where responsibility changes hands, and its purpose is to fail at the boundary rather than six hours later inside a training job that has already burned the cluster.

Four classes of assertion, in increasing cost

Validation checks are usefully sorted by what they cost to evaluate and what they can catch.

Structural checks confirm that the data parses and matches a schema: required columns present, types as declared, enumerations within their allowed set. These are linear in the number of rows and microseconds per row, and they catch the largest class of breaks: a renamed column, a changed type, a new enum value. A schema is the cheapest validation artifact a system can own, and the most frequently skipped.

Domain checks confirm that values fall in plausible ranges: a probability in $[0,1]$, a count non-negative, a timestamp not in the future, a category from a known set. They cost a comparison per value and catch unit changes and sign errors that pass a type check because the type did not change. An integer column whose values are all now ten times larger is a structural match and a domain failure.

Distributional checks compare the current partition against a reference using the statistics described in detecting distribution drift in production data. They cost a sort or a histogram per column and catch changes no per-row predicate can express, such as a feature whose mean has shifted because a filter in the upstream query changed.

Relational checks assert facts across tables: every foreign key resolves, the row count of the fact table is within a band of the previous period, the sum of a column reconciles with an upstream report. These require joins or aggregations and are the most expensive, often run as separate jobs against the built partition rather than inline.

The ordering matters because the classes are cheap-to-expensive and each catches what the cheaper ones miss. A pipeline that jumps straight to distributional monitoring will drown in alarms caused by a schema change that a ten-millisecond structural check would have identified by name.

The placement of the structural check is part of its cost model. Evaluated inside the transformation that reads the source, it stops at the first row that fails; evaluated after the pipeline has written its output, it stops after the full compute has been paid. Validating at the reader means the failure carries the source’s row identifier and the offending column, which is what the owner of the upstream table needs in order to fix it. Validating at the writer means the failure is discovered by the consumer, whose message reaches the producer through a channel that carries less context. Both placements are worth having for different reasons, and a system that validates only at its own output has moved the cost of the failure onto the next team.

Schema evolution as the primary defence

Most production data breaks are schema breaks, and schema handling differs sharply between file formats and table formats.

Columnar formats with embedded schemas, Avro, Parquet, and the table formats built on them, allow readers to reconcile an old file with a new schema. Avro’s rules are explicit: a reader schema is matched against the writer schema by name, missing fields with defaults are filled, added fields without defaults are an error, and type promotions are limited to a fixed set including int to long and float to double. That is a promotion lattice, not a free-for-all, and it is why an upstream change from int to string breaks readers where a change from int to long does not. Parquet stores per-file metadata and leaves reconciliation to the reader, so a table format above it, with its own schema history and a notion of which columns were added in which version, is what actually makes evolution safe.

Decisions about what counts as compatible are policy, not physics. Adding a nullable column is compatible in nearly every scheme. Removing a column is compatible for readers that ignore unknown fields and catastrophic for those that do not. Widening a type is compatible under Avro’s promotion rules and silently lossy if the widening is the other direction. Making a nullable column required breaks every historical file. A validation layer that encodes this policy turns a class of upstream change into a clear error at ingestion rather than a mismatch discovered during a model’s quarterly review.

The partition completeness problem

A schema can be perfect and the partition wrong. Batch pipelines are built around time partitions, and the common failure is consuming a partition before it is complete. Two rows of metadata prevent it: a row count or row-count band for the partition, and a marker written by the producing job when it finishes.

The marker is the more reliable of the two, because a partially written partition can occasionally fall within a plausible count band. The contract is that a partition is consumable only when its marker exists, and that a downstream job reading an unmarked partition treats it as absent rather than as empty. The distinction between absent and empty is precisely what a missing marker encodes, and it is lost as soon as a reader implements absent as zero rows.

Late-arriving rows complicate the marker. If the producer restates a partition after the marker is written, the marker’s meaning changes from “final” to “sealed at version $n$”. Recording the version in the marker, and in the consumer’s checkpoint, lets a restatement be detected and reprocessed deliberately rather than producing two different training sets from the same nominal inputs.

Validation as a gate, not a report

The architectural choice is whether failed checks block the pipeline or are logged for later review. Reporting is cheap and useless: a batch that trains on bad data and alerts afterwards requires a retrain, a rollback, and an explanation, and the alert’s audience has already lost the context. Blocking is expensive and correct, but only if the checks are precise enough that blocking does not become a permanent outage.

The compromise most systems adopt is a severity split. Structural and domain violations block, because they indicate the data cannot be interpreted. Distributional violations warn and quarantine: the partition is marked as unfit for training, the model continues to serve on the current version, and the partition is retained for investigation. That mapping keeps the failure contained without making every statistical fluctuation a production incident.

Two properties make the gate trustworthy. Checks are evaluated on the data actually consumed, not on a sample chosen by the pipeline, and the check results are stored alongside the dataset version so that a training run can assert that its inputs passed. A run that cannot find a passing validation record for its inputs should refuse to start; otherwise the gate is advisory no matter what the configuration says.

Distributional checks with a fixed expectation set

The distributional class is the hardest to keep stable, because a raw threshold on a drifting metric produces either constant alerts or no alerts. The technique that makes it work is to commit the expectations to source control alongside the pipeline: a file that declares, per column, the checks that must hold together with the reference statistics they are evaluated against.

expectations = {
    "order_total": {"min": 0.0, "max": 100_000.0, "null_rate_max": 0.001},
    "currency":    {"allowed": {"USD", "EUR", "GBP"}, "null_rate_max": 0.0},
    "placed_at":   {"not_future": True, "skew_max_minutes": 15},
    "order_id":    {"unique": True},
}

The values are reviewable, diffable, and changeable only by a commit that someone approved. That converts a validation suite from a notebook of ad hoc queries into an artifact with the same status as the transformation code, and it means a threshold change shows up in review rather than in a silent edit to a monitoring dashboard. The accompanying reference statistics, the mean, quantiles, and null rate the distributional checks compare against, should be regenerated from a named snapshot and stored with it, so that “the check passed last week” has a checkable meaning.

Silent corruption that no per-value check finds

The checks above operate on values and their distribution. A class of corruption leaves both intact: rows that are individually valid and jointly wrong. A join that duplicates rows, a customer with two address records fanning out an order into two rows, produces valid orders and a total count that is inflated by a factor dependent on how many duplicates exist. There is no predicate on a single row that detects it, and the distribution of each column can remain plausible.

Relational assertions catch this class: the primary key of the result must be unique, and the total must reconcile with the source. Uniqueness on the declared key is a single cheap check with a large return, and it is the check most often omitted because the table’s key is assumed to be enforced somewhere upstream. Under an append-only ingestion model with retries, uniqueness is not an upstream property; it is a property the consumer must verify.

After the data passes, the remaining question is whether the run that consumes it can be repeated exactly, which is the concern of reproducing training runs from metadata. Validation establishes that the inputs were what they claimed to be; snapshots and seeds establish that the run did the same thing twice.