Data Engineering · Cluster
Orchestration
Dependency graphs and retry semantics.
- 01 Topological Readiness in Workflow Dependency Graphs How a scheduler decides which task may start, why the dependency edge carries ordering rather than data, and what the critical path bounds.
- 02 Trigger Rules and Failure Propagation Across a Dependency Graph Why all-success is only one of several completion conditions, how per-edge skip states propagate downstream, and what an any-failed branch costs.
- 03 Idempotent Task Boundaries and Atomic Output Publication How a task makes a second execution safe, why writing to a staging location and swapping is cheaper than deduplication, and what the swap cannot protect.
- 04 Retry Policy Design: Error Classification, Backoff, and Jitter Which failures deserve another attempt, how exponential backoff and jitter change the arrival distribution, and why a retry can make an outage worse.
- 05 Durable Run State and Scheduler Reconciliation How a scheduler recovers after its own crash, why run state must be committed before a task is dispatched, and what the reconcile loop compares.
- 06 Logical Dates, Interval Boundaries, and Catchup Semantics Why a scheduled run carries a date it did not start on, how intervals are made half-open, and what happens when the schedule falls behind.
- 07 Backfill Concurrency and Collisions with the Live Schedule Why reprocessing history competes with today's run for the same slots, how partition locks arbitrate, and where the contention actually appears.
- 08 Worker Leases, Heartbeats, and the Zombie Task Problem How a lease turns liveness into a deadline, why an expired lease cannot prove a task stopped, and how fencing tokens contain the overlap.
- 09 Dynamic Task Mapping: Graphs Expanded at Run Time What changes when the number of tasks is unknown at definition time, how map indices are keyed, and why the reduce step is the hard part.
- 10 Why Exactly-Once Delivery Stops at the Workflow Boundary How end-to-end exactly-once requires three coordinated commits, which participant in a scheduled workflow cannot supply one, and what the achievable guarantee actually is.