Data Engineering · Cluster
Ingestion
Connectors, exactly-once claims, and backfill.
- 01 Cursor Columns and Where the Next Incremental Read Starts How a polling connector decides which rows are new, why a timestamp cursor loses rows at ties and clock skew, and what a composite keyset cursor fixes.
- 02 Log-Based Capture and the Write-Ahead Log as a Change Feed Why the redo log is a more faithful record of commits than any table query, how decoding turns physical records into row images, and what retention forces.
- 03 Snapshot to Stream: Establishing a Position Without Losing Writes Why a consistent snapshot and a log start point cannot be taken atomically by an unprivileged client, and the three mechanisms that close the gap.
- 04 Where the Checkpoint Goes: Offset Ordering and Two Failure Directions Committing a read position before or after the sink write chooses between losing data and repeating it, and the choice propagates through every downstream consumer.
- 05 What Exactly-Once Delivery Requires at Each Hop The guarantee is a conjunction of idempotence keys, transactional positions, and source cooperation, and it terminates at the first hop that lacks any of them.
- 06 Backfilling a Table While Its Stream Keeps Running A historical re-read competes with the live feed for the same keys, so the merge rule and the rate limit matter more than the query itself.
- 07 Schema Drift at the Connector Boundary A source that adds a column or widens a type breaks a connector in ways that depend on whether the change is read-time, write-time, or decode-time.
- 08 Split Discovery and Why a Parallel Read Is Not a Consistent Read How a connector partitions a table into independently readable ranges, and why the boundaries it discovers are only correct if the key distribution does not change.
- 09 Reconciling a Load: Counting Rows Is the Weakest Possible Check A row-count comparison agrees on many wrong tables, so reconciliation has to compare keys, then values, and needs a position to compare at.
- 10 Deletes, Tombstones, and the Append-Only Sink Object storage cannot delete a row in place, so removal becomes a recorded fact with a retention period, and every reader must agree on how to apply it.