Scientific Computing · Cluster
High-Performance Computing
Clusters, MPI, and scaling limits.
- 01 Ranks, Communicators, and the SPMD Execution Model Why a cluster program is a fixed set of processes with private address spaces, and what the communicator abstraction adds beyond a process index.
- 02 Message Matching: Tags, Ordering, and the Deadlock in Naive Exchange How a receive identifies its message, what the non-overtaking rule does and does not guarantee, and why a ring exchange deadlocks only above a size threshold.
- 03 What an Allreduce Actually Moves: Trees, Rings, and the Cost Model Why the obvious gather-reduce-scatter is the wrong shape, how ring and recursive-doubling algorithms trade latency for bandwidth, and where the two cross over.
- 04 Non-Blocking Sends, Eager Limits, and the Progress Problem Why separating initiation from completion does not automatically overlap communication with computation, and what the library must do while the application is not inside it.
- 05 Halo Exchange and the Surface-to-Volume Tax on Domain Decomposition How ghost layers turn a stencil update into communication, why block decompositions beat slabs at scale, and where the ratio stops paying.
- 06 Strong and Weak Scaling: What Amdahl and Gustafson Actually Bound The two scaling questions, the arithmetic behind their ceilings, and how to read a measured curve without attributing every bend to the algorithm.
- 07 Fat Trees, Toruses, and the Bisection Bandwidth Ceiling How the worst-case cut through a network bounds global traffic, why a torus gives up bisection bandwidth for constant degree, and where each topology fits.
- 08 NUMA First-Touch and the Memory Bandwidth Ceiling per Node Why a multi-socket node delivers less than its nominal bandwidth to an unaware program, and how page placement, binding, and the roofline interact.
- 09 Communication-Avoiding Algorithms and the Latency Floor How restructuring the arithmetic to block inner products and use fewer global synchronization steps changes the scaling exponent, and what it costs in numerical stability.
- 10 Collective Buffering and Two-Phase I/O for Parallel Checkpoints Why many ranks writing separate files is the wrong shape, how two-phase I/O regroups data into stripe-aligned chunks, and what the file system consistency rules allow.