Tag
optimizer-state
- May 25, 2026 Blockwise 8-Bit Optimizer State Without Losing the Second Moment Quantising Adam moments to eight bits cuts state memory by four, but small second-moment values quantise to zero. Blockwise scaling and dynamic exponent sharing fix that. deep learning
- Sep 13, 2024 The Training Memory Budget: Parameters, Optimizer State, Gradients, Activations How device memory is consumed during training, why a model that fits for inference can be four times too large to train, and which term to cut first. deep learning
- Sep 6, 2024 ZeRO Stages: Sharding Optimizer State, Gradients, and Parameters How the three partitioning stages divide training state across data-parallel ranks, what communication each one adds, and why stage 3 changes the residency model rather than just the arithmetic. deep learning
- Jul 4, 2024 Loss-Scale Dynamics and Spike Detection in Mixed Precision Why a fixed loss scale cannot serve both the largest and smallest gradients in a run, how dynamic scaling detects overflow, and what happens to optimizer state when a step is skipped. deep learning