Quarantine and Replay: What Happens to Records That Fail Validation

A validation rule rejects part of a batch. The rejected records are dropped, the job reports success with a row count slightly below expectation, and the rejected share is gone. The loss is not a decision anyone made; it is the default outcome of a WHERE clause that filters the failing rows out of the load.

Keeping those records instead turns one write into two and creates a second pipeline whose inputs are the first pipeline’s failures. The second pipeline has all the same problems as the first, plus a few of its own: it must preserve enough context to diagnose the failure, it must be replayable without duplicating the records that were already accepted, and it must be bounded in time, or it will accumulate every record that will never be fixable.

Rejection is a write, not a discard

A record that fails validation is information. It says that some producer emitted something outside the contract, and the shape of the failure is a signal about which producer and which part of the contract. Discarding it destroys the signal and leaves only an aggregate count, which is enough to notice that something is wrong and not enough to find out what.

Quarantining it writes the record, plus context, to a separate location. The context is the part that determines whether the quarantine is useful. At minimum it needs the rule that failed, the time of the failure, and the identity of the job that made the decision. Without the rule identifier, the quarantine is a pile of rows with an unexplained reason string, and the first investigation has to reconstruct which rule fired by re-running the validators.

The record itself must be preserved in its original form, not coerced into the target schema. A record that failed because a field had the wrong type cannot be stored in a typed column of that type, so the quarantine schema must be permissive: either the original serialised bytes with a format tag, or a schema in which every field is optional and loosely typed. Storing the raw bytes is the more faithful option and the less queryable one, since every investigation requires deserialising. Storing a permissive structured form is queryable and loses the distinctions that a strict schema would have enforced, including the one that caused the failure.

The two are not exclusive. A quarantine table can hold the parsed fields in a permissive form alongside the original payload, and the payload is what makes the record replayable after the rule or the schema is fixed.

The quarantine table needs its own contract

Quarantine tables are typically created ad hoc and then never maintained, which produces a table whose schema is a snapshot of whatever the target schema looked like when the table was created. Two failures follow.

The first is schema evolution. The target schema changes, the quarantine table does not, and a record rejected after the change cannot be written to the quarantine because its fields do not fit. The quarantine then rejects records, and the pipeline has two failure paths instead of one.

The second is the reverse: a quarantine table that tracks the target schema automatically will lose fields when the target removes them, and a record quarantined for a field that was later removed becomes unreadable in exactly the case where its content matters.

The arrangement that survives is a fixed envelope around a variable payload. The envelope holds the metadata that never changes: the ingestion time, the source identifier, the rule identifier, the payload format, and a hash of the payload. The payload holds the record, either as bytes or as a permissive structure whose schema is versioned alongside the envelope. The envelope’s stability is what allows the table to be queried across years of history, and the payload’s versioning is what allows old records to be interpreted correctly.

The payload hash is worth its bytes. It is the only field that allows a record to be recognised after it has been transformed, and it is the basis for deduplicating replays, which is the next problem.

Replay is a second write path

Replaying a quarantined record means writing it into the table it originally failed to reach. That write path has to satisfy everything the original write path satisfied, and it has an additional hazard: the record may already be present.

The record can already be present for three reasons. The validation may have rejected a record that was in fact written by a different path, such as a retry that succeeded on a second attempt while the first attempt’s failure was quarantined. The record may have been replayed once already and the replay’s success not recorded. Or the original load may have written the record and then failed on a later stage, so the record exists even though the batch as a whole is marked failed.

The defence is the same as for any at-least-once system: make the write idempotent by key, so that writing the same record twice produces the same state. That requires a key, which is the row identity declaration discussed for deduplication, and it requires the write path to apply the key rather than to append blindly.

A replay that appends without a key doubles the row count for every record it processes, and the duplication is invisible until a count is compared. A replay that overwrites by key is safe against duplication and destroys the original record if the replayed version is wrong, which is why a replay should be preceded by a check that the corrected record now passes the rule that rejected it.

-- Replay from quarantine, keyed, only for records that now pass validation.
MERGE INTO orders AS t
USING (
  SELECT order_id, customer_id, amount, event_time
  FROM quarantine.records
  WHERE rule_id = 'AMOUNT_NON_NEGATIVE'
    AND ingested_at < current_date - INTERVAL '1' DAY
    AND payload_format = 'json-v2'
) AS s
ON t.order_id = s.order_id
WHEN MATCHED AND t.amount IS DISTINCT FROM s.amount THEN UPDATE SET
  amount = s.amount, event_time = s.event_time
WHEN NOT MATCHED THEN INSERT (order_id, customer_id, amount, event_time)
  VALUES (s.order_id, s.customer_id, s.amount, s.event_time);

The IS DISTINCT FROM guard matters: without it, a merge rewrites every matched row, which produces a new file version for the whole partition and invalidates caches for records that did not change.

Ordering, and why replay is not a simple append

A record replayed late enters the table after records that describe later events. If the table is append-only and consumers read it as a sequence, the late record appears out of order, and any consumer that assumes arrival order equals event order will compute a different result from one that sorts.

For a table read by query, order does not matter, because the query imposes its own. For a stream, it does. A replay into a stream is a second producer writing into the same topic or log, and it interleaves with the live producer’s output. Consumers that aggregate over windows will see the replayed record in a later window than its event time, and if the window has already closed, the record is late by the consumer’s own definition and is dropped or handled by the late-data policy.

This is the same tension that freshness measured from event time expresses for batch tables, and it has a specific consequence for replay design: a replay that must be correct for a windowed consumer has to be delivered in event-time order, or the consumer’s windows have to be reopened. Reopening a window is a rewrite of its output; delivering in order requires buffering the replay until it can be interleaved, which is only possible if the replay’s event times are known and bounded.

For batch tables the problem is milder and still present. A partition rewritten by a replay is a new version, and a consumer that read the old version and cached it will not see the change. The staleness is bounded by the consumer’s cache lifetime, which is a property of the consumer rather than of the table and is therefore invisible to anything the pipeline records.

Bounding the quarantine

A quarantine that is never drained grows without bound, and its growth is dominated by records that will never be fixable: those from a producer that has been decommissioned, those whose failure was a legitimate rejection of invalid input that no rule change will admit, and those whose context is insufficient to diagnose anything.

Three mechanisms keep it bounded, and each requires a decision that must be made explicitly.

An expiry, after which a record is deleted. The expiry has to be longer than the time it takes to fix a rule, which is a statement about the organisation rather than the data. It also has to be shorter than the time after which the record’s context is no longer reproducible, since replaying a record against a schema that has moved on twice may produce a different result from what it would have produced at the time.

A classification, so that records are separated by whether they are actionable. A record that failed a rule which has since been relaxed is replayable. A record that failed because a required field was absent and the producer never supplies it is not, unless the field becomes optional. Grouping by rule identifier and by source gives the counts that make the classification possible, and it is the query that should be run first on any quarantine table.

A cap, beyond which new records are sampled rather than all retained. Sampling preserves the diagnostic signal, which is about rates and shapes, while bounding the storage. It loses the ability to replay the specific record that a downstream consumer is missing, which is the case where a full retention matters. The compromise is to retain all records for a recent window and a sample beyond it, so that the recent failures, which are the ones most likely to be replayed, are complete.

What the quarantine measures

The counts in a quarantine table are a data quality metric, and they measure something different from what a validation pass rate measures.

A pass rate is a ratio, and a ratio can be stable while the population underneath it changes. A pass rate of ninety-eight percent over ten million records is two hundred thousand failures, and the same rate over a hundred thousand records is two thousand. The rate says the contract is being met at the same frequency; the count says the volume of work the quarantine represents.

The more informative breakdown is by rule and by source. A single rule dominating the failures means the rule is either wrong or the source is producing a specific malformation, and the two are distinguished by whether the rule’s failure rate is stable over time. A single source dominating means that source’s contract has drifted, which is the case that a per-source schema contract exists to prevent.

The metric that is usually missing is time to drain: how long a quarantined record waits before it is replayed or expired. That figure is the quarantine’s actual latency, and it is the one that determines whether a consumer downstream of the table is missing data for hours or for weeks. It is computable from the envelope’s ingestion time and the replay’s commit time, and it is the number worth putting next to the pass rate, because a pass rate of ninety-nine percent with a drain time of a month is a different system from the same rate with a drain time of an hour.

The cases where rejecting is correct

Not every rejection should be reversible. A record that violates a constraint because the source is genuinely producing invalid data should be rejected and the source should be fixed, because replaying it would admit data that the constraint exists to exclude. A record whose failure indicates a corrupted payload cannot be repaired by a rule change and will fail again on replay.

Distinguishing these from the fixable cases requires knowing why the rule exists, which is metadata that most rule definitions do not carry. A rule with a stated rationale, and a stated disposition for its failures, can be classified automatically; a rule without one has to be classified by a person each time it fires.

The disposition is the useful part: a rule can be declared as rejecting (the record is invalid and should be dropped), quarantining (the record is suspect and should be held), or annotating (the record is admitted with a flag). The third option is the one most often overlooked and the one that loses the least: a record that fails a soft check, such as an implausible but possible value, can be written with the failure recorded alongside it, so that downstream consumers can choose whether to exclude it. That converts a binary accept-reject decision into a filter that each consumer applies according to its own tolerance, and it is the only arrangement that serves consumers with different requirements from the same table.