Distributed Systems · Cluster
Operating Distributed Systems
Rollouts, backfills, and recovery.
- 01 Rollout Waves and the Arithmetic of Blast Radius How staged deployment divides one risky decision into a sequence of gated ones, why wave size sets the smallest detectable failure rate, and where correlated faults break the model.
- 02 How Many Requests a Canary Gate Needs Before Its Verdict Means Anything The sample-size arithmetic behind canary health checks, why rare failures and tail latencies need orders of magnitude more traffic, and how sequential testing changes the threshold.
- 03 Mixed-Version Operation: What Two Versions Must Agree On Why every rolling upgrade temporarily runs two builds at once, which interfaces must be bidirectional across that window, and how expand-migrate-contract keeps them so.
- 04 Resumable Backfills: Watermarks, Chunk Boundaries, and Gap-Free Progress How a long-running backfill records where it stopped, why the watermark must advance over a total order rather than a clock, and what idempotent chunks cost.
- 05 Admission Control for Background Work: Keeping Scans Off the Critical Path Why a fixed batch size cannot protect foreground latency, how a feedback controller turns saturation signals into a work rate, and where the loop becomes unstable.
- 06 Verifying a Backfill with Range Checksums Instead of Full Comparison How partitioning a dataset into key ranges and comparing per-range digests localises a skipped or double-written record, and why the digest must be order-independent.
- 07 Rolling Back a Change That Already Mutated State Why reverting a binary does not revert its writes, which mutations admit a compensating action, and how to decide before deployment whether a rollback path exists at all.
- 08 Level-Triggered Reconciliation: Comparing State Instead of Consuming Events Why a controller that reacts to change events loses convergence after a missed event, how periodic full comparison restores it, and what the comparison costs at scale.
- 09 Cold Start After a Full Outage: Recovery Order and the Thundering Herd Why restarting a whole system at once produces failures that no individual component has, how dependency order and admission shape recovery, and where warmup ends.
- 10 Cutover Between Two Systems: Dual Writes, Shadow Reads, and the Switchover Point How a migration moves authority from one store to another without a window of divergent writes, and why the cutover must be a single atomic epoch change.