Data Engineering · Cluster
Data Quality
Validation, lineage, and anomaly detection.
- 01 Where a Schema Contract Is Enforced, and What It Cannot Catch Write-time validation against a declared schema, read-time inference, and the class of errors that survives both: structural checks constrain types, not meanings.
- 02 Row Identity and Why Deduplication Needs a Key, Not a Comparison Deciding that two records describe the same entity requires a declared key; comparing whole rows fails on late-arriving corrections, and windowed dedup only moves the ambiguity.
- 03 Measuring Freshness From Event Time, Not Arrival Time A table can be loaded on schedule and still be stale: lag is a distribution over records, and comparing the newest row timestamp to the clock hides the tail.
- 04 How Null Rates and Distinct Counts Detect a Broken Join Column profiles are cheap aggregates that catch errors no type check can: a null-rate jump or a collapsed distinct count localises a fault to one column.
- 05 Partition Statistics as Validation Signals Per-partition row counts, byte sizes, and file counts are metadata, so checking them costs almost nothing; the hard part is knowing which deviation means which fault.
- 06 Column-Level Lineage and the Cost of Tracking It Deriving which upstream column feeds which downstream column requires parsing every query; the alternative is coarse and the fine-grained version has an unbounded cost.
- 07 Seasonal Baselines and the False Alerts They Prevent A fixed threshold on a metric that varies by weekday produces a stream of alerts that gets ignored; a baseline has to model the periodicity it is compared against.
- 08 Orphan Keys: Checking Referential Integrity Without Foreign Keys Warehouses rarely enforce foreign keys, so referential integrity becomes a scheduled anti-join; its cost, its two directions, and the time dimension that makes it ambiguous.
- 09 Two-Sample Tests for Distribution Drift in a Column A mean and a count can hold steady while a column reshapes itself; comparing histograms detects that, but the test, the bins, and the sample size all change the answer.
- 10 Quarantine and Replay: What Happens to Records That Fail Validation Rejecting a record is a write to a second table, and replaying it later is a second write path; both need a schema, an ordering rule, and a bound on how long they stay open.