Machine Learning · Cluster
Optimization
Gradients, constraints, and convergence.
- 01 Step Sizes and the Descent Lemma in Gradient Descent How the Lipschitz constant of the gradient bounds a usable step size, why 1/L is the natural choice for quadratic objectives, and which constants the convergence rates hide.
- 02 Gradient Noise, Batch Size, and Step Decay in Stochastic Gradient Descent Why sampling a subset of the data makes the gradient unbiased but noisy, how that noise sets a floor under constant step sizes, and what step schedules can and cannot fix.
- 03 Momentum and Nesterov Acceleration in Gradient Descent How velocity terms turn the gradient sequence into a filtered signal, why the quadratic case admits a closed-form optimal damping, and what breaks when gradients are noisy or the objective is not convex.
- 04 Adaptive Step Sizes: AdaGrad, RMSProp, and Adam How per-coordinate accumulators of squared gradients rescale the update, why the resulting methods needed repair to keep their convergence guarantees, and what the state costs.
- 05 Lagrange Duality and the KKT Conditions in Constrained Learning How multipliers convert constrained problems into stationary points of an unconstrained function, when the dual problem has the same value as the primal, and where these conditions stop being sufficient.
- 06 Projected and Proximal Gradient Methods for Composite Objectives How constraint sets and non-differentiable penalties are handled by composing a gradient step with a projection or a proximal operator, and what the convergence rates assume.
- 07 Quasi-Newton Methods and the L-BFGS Two-Loop Recursion How secant equations build a Hessian approximation from successive gradient differences, why the limited-memory form stores only a few vector pairs, and how line searches keep the approximation positive definite.
- 08 Variance Reduction in Stochastic Gradient Descent: SVRG and SAGA How control variates make the stochastic gradient noise vanish near the solution, why that restores a linear rate for finite sums, and what the stored state and epoch length cost.
- 09 Escaping Saddle Points in Nonconvex Optimization Why a vanishing gradient is not evidence of a minimum in high dimensions, how negative curvature is detected and followed, and what the escape guarantees cost per iteration.
- 10 Parallel and Asynchronous SGD: Staleness and Convergence Bounds How splitting a gradient computation across workers changes the update rule, why delayed gradients degrade the guarantee in proportion to the delay, and what communication efficiency costs in accuracy.