Tag
collectives
- Jun 5, 2026 What an Allreduce Actually Moves: Trees, Rings, and the Cost Model Why the obvious gather-reduce-scatter is the wrong shape, how ring and recursive-doubling algorithms trade latency for bandwidth, and where the two cross over. scientific computing
- May 29, 2025 Tensor-Parallel Residency and Interconnect Pressure Sharding a model across devices splits weights and key-value heads, but every decode step becomes a collective; how that changes the memory and latency balance. inference systems