Tag
gradient-noise
- Jul 3, 2025 Momentum and Nesterov Acceleration in Gradient Descent How velocity terms turn the gradient sequence into a filtered signal, why the quadratic case admits a closed-form optimal damping, and what breaks when gradients are noisy or the objective is not convex. machine learning
- Jul 6, 2023 Gradient Noise, Batch Size, and Step Decay in Stochastic Gradient Descent Why sampling a subset of the data makes the gradient unbiased but noisy, how that noise sets a floor under constant step sizes, and what step schedules can and cannot fix. machine learning