Tag
step-size
- Sep 5, 2026 Adaptive Step Sizes: AdaGrad, RMSProp, and Adam How per-coordinate accumulators of squared gradients rescale the update, why the resulting methods needed repair to keep their convergence guarantees, and what the state costs. machine learning
- Aug 15, 2025 Step Sizes and the Descent Lemma in Gradient Descent How the Lipschitz constant of the gradient bounds a usable step size, why 1/L is the natural choice for quadratic objectives, and which constants the convergence rates hide. machine learning
- Jul 6, 2023 Gradient Noise, Batch Size, and Step Decay in Stochastic Gradient Descent Why sampling a subset of the data makes the gradient unbiased but noisy, how that noise sets a floor under constant step sizes, and what step schedules can and cannot fix. machine learning