Tag
adaptive-optimizers
- Feb 27, 2026 Decoupled Weight Decay and Adaptive Scaling Why adding a shrinkage term to the gradient is not the same as shrinking the parameter, how the adaptive denominator changes the effective decay per coordinate, and what the decoupled update fixes. deep learning
- Feb 17, 2024 Learning Rate Warmup and the Early Instability Window Why the first few hundred optimization steps are different from the rest, what large early updates do to adaptive optimizers, and how warmup bounds the damage. deep learning