Tag
convergence-rates
- May 4, 2026 Variance Reduction in Stochastic Gradient Descent: SVRG and SAGA How control variates make the stochastic gradient noise vanish near the solution, why that restores a linear rate for finite sums, and what the stored state and epoch length cost. machine learning
- Aug 15, 2025 Step Sizes and the Descent Lemma in Gradient Descent How the Lipschitz constant of the gradient bounds a usable step size, why 1/L is the natural choice for quadratic objectives, and which constants the convergence rates hide. machine learning
- Jul 3, 2025 Momentum and Nesterov Acceleration in Gradient Descent How velocity terms turn the gradient sequence into a filtered signal, why the quadratic case admits a closed-form optimal damping, and what breaks when gradients are noisy or the objective is not convex. machine learning
- Apr 5, 2025 Parallel and Asynchronous SGD: Staleness and Convergence Bounds How splitting a gradient computation across workers changes the update rule, why delayed gradients degrade the guarantee in proportion to the delay, and what communication efficiency costs in accuracy. machine learning
- Jun 19, 2024 Escaping Saddle Points in Nonconvex Optimization Why a vanishing gradient is not evidence of a minimum in high dimensions, how negative curvature is detected and followed, and what the escape guarantees cost per iteration. machine learning
- Jul 6, 2023 Gradient Noise, Batch Size, and Step Decay in Stochastic Gradient Descent Why sampling a subset of the data makes the gradient unbiased but noisy, how that noise sets a floor under constant step sizes, and what step schedules can and cannot fix. machine learning
- Mar 12, 2023 Projected and Proximal Gradient Methods for Composite Objectives How constraint sets and non-differentiable penalties are handled by composing a gradient step with a projection or a proximal operator, and what the convergence rates assume. machine learning