Tag
adam
- Sep 5, 2026 Adaptive Step Sizes: AdaGrad, RMSProp, and Adam How per-coordinate accumulators of squared gradients rescale the update, why the resulting methods needed repair to keep their convergence guarantees, and what the state costs. machine learning
- May 25, 2026 Blockwise 8-Bit Optimizer State Without Losing the Second Moment Quantising Adam moments to eight bits cuts state memory by four, but small second-moment values quantise to zero. Blockwise scaling and dynamic exponent sharing fix that. deep learning