Tag
learning-rate-schedules
- Oct 4, 2024 Cosine and Linear Decay Schedules as Step-Size Integrals What a learning rate schedule controls beyond the final value, how the shape distributes total parameter displacement across training, and why the choice between linear and cosine changes little at equal area. deep learning
- Feb 17, 2024 Learning Rate Warmup and the Early Instability Window Why the first few hundred optimization steps are different from the rest, what large early updates do to adaptive optimizers, and how warmup bounds the damage. deep learning