Topic domain
Distributed Systems
Coordination and correctness across unreliable networks.
- Articles
- 80
- Clusters
- 8
- Foundational
- 24
- Intermediate
- 32
- Advanced
- 24
Clocks, causality, and happens-before.
- Wall-Clock Timestamps Do Not Order Distributed Events
- Happens-Before and the Scalar Logical Clock
- Vector Clocks and the Detection of Concurrency
- Interval Tree Clocks and Membership Change
- The NTP Four-Timestamp Exchange and Its Asymmetry Limit
- Monotonic and Civil Clocks in Running Programs
Replication
10Consistency models and replica placement.
- Replication Topologies and What Each One Tolerates
- Synchronous and Asynchronous Replication: Where the Commit Point Sits
- Leader Leases and Fencing Tokens in Failover
- Version Vectors and Conflict Detection in Multi-Writer Replication
- Read Repair and Hinted Handoff on the Write Path
- Quorum Intersection and the Arithmetic of Stale Reads
Consensus
10Agreement protocols and their failure assumptions.
- Why Majority Quorums Intersect and What the Intersection Buys
- FLP Impossibility and Why Real Protocols Assume Partial Synchrony
- Raft Elections: Terms, Votes, and the Pre-Vote Correction
- Paxos Prepare and Accept: What the Promises Are For
- Commit Rules, Figure 8, and Why a Leader Cannot Trust Its Log
- Membership Change Under Joint Consensus
Partitioning
10Sharding, rebalancing, and hot spots.
- Hash Partitioning and the Range Query It Destroys
- Consistent Hashing Rings and Virtual Node Counts
- Range Partition Boundaries and the Split Key Contract
- Shard Migration Cutover and the Dual-Write Window
- Why Append-Only Keys Concentrate Load on One Shard
- Routing Metadata Staleness and the Cost of a Wrong Guess
Atomic commit and isolation across nodes.
- Two-Phase Commit and the Coordinator Decision Record
- Presumed Abort and Presumed Commit: Two Defaults for Missing Decisions
- In-Doubt Transactions, Held Locks, and Why Timeouts Cannot Help
- What Three-Phase Commit Adds and What It Still Cannot Do
- Detecting Deadlock Across Nodes Without a Shared Wait-For Graph
- Global Snapshots Across Shards Without a Shared Clock
Heartbeats, suspicion, and partial failure.
- Completeness and Accuracy: The Two Properties Every Failure Detector Trades Against
- Heartbeat Interval and Timeout Arithmetic for Fixed-Threshold Detectors
- SWIM Ping-Request Probes and Why Suspicion Beats Immediate Declaration
- Phi Accrual Detectors: Turning Inter-Arrival Samples into a Continuous Suspicion Level
- TCP Keepalive as a Failure Detector, and Why It Cannot Be One
- Disseminating Suspicion by Gossip: Anti-Entropy, Rumor Mongering, and Merge Order
Leases, locks, and membership.
- The Lease as a Time-Bounded Grant: Why Expiry Needs No Message
- The Linearizable Register Beneath Every Coordination Primitive
- Sessions and Ephemeral Nodes: Tying State to a Live Connection
- Compare-and-Swap Retry Loops and the ABA Problem on a Versioned Register
- One-Shot Watches and the Changes That Fall Between Them
- Fair Lock Queues from Sequential Nodes
Rollouts, backfills, and recovery.
- Rollout Waves and the Arithmetic of Blast Radius
- How Many Requests a Canary Gate Needs Before Its Verdict Means Anything
- Mixed-Version Operation: What Two Versions Must Agree On
- Resumable Backfills: Watermarks, Chunk Boundaries, and Gap-Free Progress
- Admission Control for Background Work: Keeping Scans Off the Critical Path
- Verifying a Backfill with Range Checksums Instead of Full Comparison
All Distributed Systems articles
- Oct 19, 2026 Leaderless Ordering: Dependency Graphs Instead of a Single Log How per-command sequence numbers and dependency sets let replicas commit concurrently, why conflicts still impose an order, and where recovery becomes the expensive part.
- Oct 13, 2026 Why Append-Only Keys Concentrate Load on One Shard Monotonic keys place every insert at the end of the key space, so one partition absorbs the write traffic. The fixes, from bit-reversal to bucket suffixes, and what each costs.
- Oct 2, 2026 Bootstrapping a Replica from a Snapshot and a Log Tail Why a running replica cannot simply be copied file by file, how a consistent cut is captured while writes continue, and how the log position inside the snapshot stitches it to the stream.
- Aug 15, 2026 Wall-Clock Timestamps Do Not Order Distributed Events Why two machines reading the same instant produce different numbers, how far the disagreement grows between synchronizations, and what an interval-bounded timestamp can still support.
- Aug 14, 2026 Verifying a Backfill with Range Checksums Instead of Full Comparison How partitioning a dataset into key ranges and comparing per-range digests localises a skipped or double-written record, and why the digest must be order-independent.
- Jul 29, 2026 Leader Leases and Fencing Tokens in Failover Why a demoted leader can still issue writes, how time-based leases bound that window, and why fencing tokens rather than timeouts are what actually protects shared storage.
- Jul 27, 2026 Two-Phase Commit and the Coordinator Decision Record Why a distributed commit needs a durable, single-source decision, what each participant promises when it votes yes, and how the coordinator log resolves a crash in the middle of the protocol.
- Jul 23, 2026 Fair Lock Queues from Sequential Nodes How a ticket built from a sequential child name turns lock contention into a queue, why each waiter watches only its predecessor, and what a herd costs by comparison.
- Jul 17, 2026 How Many Requests a Canary Gate Needs Before Its Verdict Means Anything The sample-size arithmetic behind canary health checks, why rare failures and tail latencies need orders of magnitude more traffic, and how sequential testing changes the threshold.
- Jul 16, 2026 SWIM Ping-Request Probes and Why Suspicion Beats Immediate Declaration How a node delegates a probe to a third party before declaring a peer dead, and how incarnation numbers let a wrongly suspected node refute the claim without a restart.
- Jun 24, 2026 Mixed-Version Operation: What Two Versions Must Agree On Why every rolling upgrade temporarily runs two builds at once, which interfaces must be bidirectional across that window, and how expand-migrate-contract keeps them so.
- Jun 10, 2026 Serializable Snapshot Isolation When Reads Span Several Shards How SSI tracks rw-dependencies to detect dangerous structures, why the conflict graph must be shared between shards, and what the read-only optimisation costs once reads are remote.
- Jun 2, 2026 Phi Accrual Detectors: Turning Inter-Arrival Samples into a Continuous Suspicion Level How a detector fits a distribution to observed heartbeat gaps, converts the current gap into a probability that the peer is dead, and why the exponential form of that probability is what makes a threshold portable.
- May 27, 2026 Splitting a Partition While Writes Continue The in-place split protocol: freeze, cut, publish, then let readers discover the new boundary. Why the split point is a durable record and why the parent must keep serving.
- May 25, 2026 Replicating the Commit Decision So the Coordinator Can Die Why a single durable decision record is a single point of failure, how consensus turns commit into a replicated log entry, and what the extra round trip buys in recovery time.
- May 15, 2026 One-Shot Watches and the Changes That Fall Between Them How watch registration, firing, and re-registration create a gap, why the client must re-read rather than trust the event, and what a lost notification actually means.
- May 11, 2026 Version Vectors and Conflict Detection in Multi-Writer Replication How per-replica counters encode causal history, why vector comparison identifies concurrency that timestamps hide, and what the vectors cost in metadata and pruning.
- May 5, 2026 Total-Order Broadcast: One Order for Every Replica Why a single sequencer turns concurrent updates into a deterministic log, what the ordering guarantee costs in latency and availability, and how the order survives a sequencer failure.
- May 4, 2026 Lease Safety Margins Under Clock Drift The arithmetic that keeps a lease from being granted twice: how offset error, drift rate, and timer resolution set the gap between the client deadline and the server reassignment.
- Apr 27, 2026 Commit Rules, Figure 8, and Why a Leader Cannot Trust Its Log Why Raft only commits entries at a term boundary, the reordered-commit scenario that defeats the naive rule, and how the commit index interacts with log repair.
- Mar 23, 2026 Consistent Hashing Rings and Virtual Node Counts Why a ring moves only 1/(n+1) of the keys when a node joins, how virtual nodes fix the variance, and what the load spread looks like at realistic counts.
- Feb 27, 2026 Merkle Tree Anti-Entropy: Finding Divergence Without Comparing Everything How hash trees over key ranges reduce replica comparison from a full data exchange to a walk over the subtrees that actually differ, and what write churn does to that saving.
- Feb 10, 2026 Why Majority Quorums Intersect and What the Intersection Buys The arithmetic of quorum overlap, why read and write sets must share a node, and what asymmetric quorum sizes cost in latency, durability, and fault tolerance.
- Jan 30, 2026 The Linearizable Register Beneath Every Coordination Primitive Why locks, membership lists, and leader election all reduce to reads and writes of one small object, and what linearizability adds that sequential consistency does not.
- Jan 26, 2026 The NTP Four-Timestamp Exchange and Its Asymmetry Limit How four timestamps yield an offset estimate, why the estimate is only as good as the path symmetry, and how the protocol decides which of several servers to believe.
- Jan 22, 2026 Bounded Clock Uncertainty and the Commit-Wait Rule How a clock that returns an interval instead of an instant lets a system order transactions by physical time, why reads must wait out the uncertainty, and what the wait costs.
- Jan 22, 2026 Sagas, Compensations, and the Semantics a Rollback Cannot Restore How long-running distributed transactions replace atomic rollback with forward recovery, why a compensation is not an inverse, and what the scheduler must guarantee to make it work.
- Jan 12, 2026 Consistent Cuts and the Marker Algorithm for Global Snapshots Why a set of per-process snapshots taken at different moments is usually inconsistent, and how markers travelling on the same channels as messages capture a cut that could have occurred.
- Jan 5, 2026 Global Snapshots Across Shards Without a Shared Clock How a transaction gets a consistent view of data spread over independent nodes, what a snapshot identifier must encode, and why the barrier between shards costs a round trip per query.
- Dec 30, 2025 Completeness and Accuracy: The Two Properties Every Failure Detector Trades Against Why a detector that never mistakes a live process for a dead one must sometimes wait forever, and how the completeness and accuracy properties make that trade precise.
- Nov 25, 2025 Raft Elections: Terms, Votes, and the Pre-Vote Correction How terms impose a total order on leadership attempts, why the voted-for rule plus the log-suffix check prevents two leaders in one term, and what pre-vote and check-quorum add.
- Nov 25, 2025 Replication Topologies and What Each One Tolerates How single-leader, multi-leader, and leaderless replication differ in where writes are accepted, which node losses they survive, and what conflict handling each one forces.
- Nov 22, 2025 Monotonic and Civil Clocks in Running Programs Why interval measurement needs a clock that never goes backwards, how leap seconds and suspend break civil time, and which timeout a kernel or runtime actually uses.
- Nov 14, 2025 Happens-Before and the Scalar Logical Clock How Lamport replaced physical time with a partial order defined by message passing, why one integer per process suffices for that order, and what the integer provably cannot tell you.
- Oct 28, 2025 Sessions and Ephemeral Nodes: Tying State to a Live Connection How a coordination service tracks client liveness, why ephemeral nodes vanish with the session, and what a client must do when it cannot tell whether its session survived.
- Oct 24, 2025 Timestamp Oracles and Global Transaction Order How distributed systems assign a total order to transactions without a shared clock, what happens when the oracle is unreachable, and why the ordering guarantee is not the same as real-time order.
- Oct 15, 2025 Local and Global Secondary Indexes Under Partitioning An index can be partitioned with the data or by the indexed value. The first makes writes cheap and reads expensive; the second inverts it, and neither is free.
- Sep 18, 2025 Why Locks Across Independent Services Do Not Compose Two lock services with no shared order can each grant a lock that the other believes excludes, and no client-side protocol repairs the missing global order.
- Sep 12, 2025 Correlated Timeouts and Detection Storms When Every Monitor Sees the Same Fault Why independent observers are only independent when their failures are uncorrelated, how a shared cause turns many weak suspicions into a confident wrong verdict, and which randomization actually helps.
- Aug 22, 2025 Rebalance Storms and the Backpressure That Prevents Them When a node fails, every partition it held becomes a migration at once. Why the recovery traffic can exceed the cluster bandwidth, and how admission control keeps the two from feeding each other.
- Jul 19, 2025 Detecting Deadlock Across Nodes Without a Shared Wait-For Graph How distributed deadlock detection assembles wait-for edges from independent lock managers, why stale edges produce phantom cycles, and what a timeout costs instead.
- Jul 15, 2025 Stop-the-World Pauses and the False Suspicion They Cause Why a collector pause, a page-fault storm, or a hypervisor deschedule looks identical to a crash from outside, and how a process can tell the difference using only local evidence.
- Jul 11, 2025 Heartbeat Interval and Timeout Arithmetic for Fixed-Threshold Detectors How the choice of heartbeat period and missed-beat threshold sets false-positive rate, detection latency, and message volume at the same time, and why the tail of the delay distribution decides everything.
- Jun 19, 2025 Rolling Back a Change That Already Mutated State Why reverting a binary does not revert its writes, which mutations admit a compensating action, and how to decide before deployment whether a rollback path exists at all.
- Jun 13, 2025 Cutover Between Two Systems: Dual Writes, Shadow Reads, and the Switchover Point How a migration moves authority from one store to another without a window of divergent writes, and why the cutover must be a single atomic epoch change.
- Jun 11, 2025 The Lease as a Time-Bounded Grant: Why Expiry Needs No Message How a lease replaces an explicit release with a deadline, why the holder must stop before the server reassigns, and what the gap between the two clocks costs.
- Apr 28, 2025 Hierarchical Heartbeat Aggregation and the Cost of Intermediate Levels Why a tree of monitors cuts liveness traffic from quadratic to linear, what the extra level adds to detection time, and how the aggregate view loses the identity of the observer.
- Apr 22, 2025 In-Doubt Transactions, Held Locks, and Why Timeouts Cannot Help The prepared-but-undecided state that two-phase commit creates, why a participant cannot resolve it locally, and how heuristics and lock timeouts trade atomicity for availability.
- Apr 15, 2025 Failure-Domain Placement: Keeping Replicas Out of the Same Blast Radius How rack, host, and power topology constrain where the third replica goes, why capacity-weighted placement and CRUSH avoid a central allocator, and how correlated loss is measured.
- Apr 7, 2025 Level-Triggered Reconciliation: Comparing State Instead of Consuming Events Why a controller that reacts to change events loses convergence after a missed event, how periodic full comparison restores it, and what the comparison costs at scale.
- Apr 3, 2025 Interval Tree Clocks and Membership Change How a tree-shaped stamp splits and joins without renumbering every process, why vector clocks break when the participant set changes, and what the fork and join operations cost.
- Mar 26, 2025 What Three-Phase Commit Adds and What It Still Cannot Do The pre-commit phase that removes blocking under node crashes, the assumption about network partitions that it silently depends on, and why real systems rarely deploy it.
- Mar 26, 2025 Why Byzantine Fault Tolerance Costs a Third of the Cluster The 3f+1 bound, where it comes from, and how quorum sizes must change when a faulty node can lie rather than merely stop.
- Mar 19, 2025 Resumable Backfills: Watermarks, Chunk Boundaries, and Gap-Free Progress How a long-running backfill records where it stopped, why the watermark must advance over a total order rather than a clock, and what idempotent chunks cost.
- Mar 17, 2025 TCP Keepalive as a Failure Detector, and Why It Cannot Be One What keepalive probes actually prove about a peer, why the kernel default of two hours makes them useless for failover, and which failures they detect that an application heartbeat cannot.
- Jan 28, 2025 Fencing Tokens: Why a Lock Is Advisory Until the Resource Checks It How a monotonic token attached to a grant lets the protected resource reject a stale holder, and why no amount of clock discipline makes the check unnecessary.
- Dec 24, 2024 Read Index and Leader Leases: Serving Reads Without a Log Entry Why a committed write is not yet a readable write, how a quorum heartbeat round establishes read freshness, and what a lease assumes about clocks.
- Dec 20, 2024 Shard Migration Cutover and the Dual-Write Window How a partition moves between machines without losing writes, why the cutover is a state machine rather than a copy, and what has to be true before the source is retired.
- Dec 12, 2024 Vector Clocks and the Detection of Concurrency Why one counter per process turns the happens-before relation into a comparison, how the merge rule is defined, and the per-message size that makes the scheme expensive at scale.
- Nov 15, 2024 Read Repair and Hinted Handoff on the Write Path How leaderless stores bring lagging replicas forward without a full scan, why hinted handoff is a temporary buffer rather than a repair, and where each mechanism fails under sustained failure.
- Oct 10, 2024 Disseminating Suspicion by Gossip: Anti-Entropy, Rumor Mongering, and Merge Order How a suspicion reaches every node without a broadcast tree, why the merge rule must be a total order, and what the log factor in convergence time actually depends on.
- Aug 30, 2024 Hybrid Logical Clocks: Physical Time With a Logical Counter How HLC timestamps stay close to wall-clock time while preserving causality, why the counter only appears inside the uncertainty window, and what a bounded timestamp costs.
- Aug 22, 2024 Paxos Prepare and Accept: What the Promises Are For Why acceptors must promise not to accept older proposals, how the highest-numbered proposal wins a phase, and what breaks when the two phases are collapsed.
- Aug 16, 2024 Membership Change Under Joint Consensus Why a configuration cannot be edited in place, how C_old,new requires dual majorities during the transition, and what single-server changes cost in safety margin.
- Jul 21, 2024 Rollout Waves and the Arithmetic of Blast Radius How staged deployment divides one risky decision into a sequence of gated ones, why wave size sets the smallest detectable failure rate, and where correlated faults break the model.
- May 6, 2024 Session Guarantees: Read-Your-Writes and Monotonic Reads How four per-session guarantees are implemented with version tracking, why they are cheaper than global consistency, and where client redirection breaks them.
- Feb 27, 2024 Compare-and-Swap Retry Loops and the ABA Problem on a Versioned Register Why an optimistic update loop can silently apply a stale decision, how version numbers close the window, and what the loop costs when contention is high.
- Feb 2, 2024 Presumed Abort and Presumed Commit: Two Defaults for Missing Decisions How a participant resolves a transaction when no decision record exists anywhere, what each presumption costs in log writes and retained state, and why presumed abort dominates in practice.
- Dec 12, 2023 Admission Control for Background Work: Keeping Scans Off the Critical Path Why a fixed batch size cannot protect foreground latency, how a feedback controller turns saturation signals into a work rate, and where the loop becomes unstable.
- Dec 11, 2023 Quorum Intersection and the Arithmetic of Stale Reads Why R + W > N guarantees a read sees the latest acknowledged write, what that inequality does not guarantee, and how sloppy quorums and read-your-writes routing break it.
- Nov 22, 2023 Coarse-Grained Locking by Design: What a Chubby Cell Optimizes For Why a lock service built for locks held for hours makes different choices than one built for request-path coordination, and where those choices stop paying.
- Sep 27, 2023 Synchronous and Asynchronous Replication: Where the Commit Point Sits How the moment a write is acknowledged determines durability, latency, and the amount of data a failover can lose, and why semi-synchronous replication is the usual compromise.
- Sep 25, 2023 FLP Impossibility and Why Real Protocols Assume Partial Synchrony The bivalent-configuration argument behind the FLP result, why the proof needs only one faulty process, and how partial synchrony and randomized termination step around it.
- Apr 29, 2023 Routing Metadata Staleness and the Cost of a Wrong Guess Every client caches a map from key to node, and every cache is wrong eventually. How epochs, redirects, and fencing turn a stale guess into a retry instead of a lost write.
- Mar 30, 2023 Cold Start After a Full Outage: Recovery Order and the Thundering Herd Why restarting a whole system at once produces failures that no individual component has, how dependency order and admission shape recovery, and where warmup ends.
- Mar 2, 2023 Range Partition Boundaries and the Split Key Contract How ordered partitions are divided, why a split key is a durable fact rather than a hint, and what happens when boundaries are chosen badly.
- Feb 28, 2023 Snapshot Install and the Cost of Compacting a Consensus Log How snapshots bound log growth, why installing a snapshot is a different operation from sending entries, and where the boundary index and chunking interact with elections.
- Jan 19, 2023 Asymmetric Partitions and What a Set of Conflicting Suspicion Reports Means How to read a matrix of who can reach whom, why a single suspicion report cannot distinguish a dead peer from a broken path, and what a system can safely conclude from per-observer evidence.
- Jul 31, 2022 Scatter-Gather Fan-Out and Tail Latency Multiplication A query over P partitions waits for the slowest one, so the 99th percentile of a single node becomes the median of the cluster. How hedged requests and partial results change the curve.
- Apr 20, 2022 Hash Partitioning and the Range Query It Destroys How modulo hashing places keys on nodes without any directory, why the same function makes ordered access impossible, and what a resize really moves.