Tag
data-parallelism
- May 1, 2026 Ring All-Reduce and the Cost of Averaging Gradients Across Devices Why the ring algorithm moves a fixed multiple of the gradient size regardless of device count, and how bucketing hides the latency that the bandwidth term does not explain. deep learning
- Oct 26, 2025 Fixed-Width SIMD Lanes Versus Scalable Vector Registers Why one design bakes the vector length into the instruction encoding and the other reports it at run time, and how each choice propagates into code generation, binary compatibility, and register pressure. computer architecture
- Sep 6, 2024 ZeRO Stages: Sharding Optimizer State, Gradients, and Parameters How the three partitioning stages divide training state across data-parallel ranks, what communication each one adds, and why stage 3 changes the residency model rather than just the arithmetic. deep learning