Deep Learning · Cluster
Training Dynamics
Initialization, normalization, and schedules.
- 01 Variance Preservation in Weight Initialization How the Xavier and He scaling factors are derived from the fan-in of a layer, what breaks when the constant is wrong, and why depth turns a small gain error into a vanished signal.
- 02 Batch Normalization and the Train-Eval Gap Why batch statistics at training time and running averages at inference time produce different outputs, and how that mismatch is exploited to estimate what a network actually learned.
- 03 Learning Rate Warmup and the Early Instability Window Why the first few hundred optimization steps are different from the rest, what large early updates do to adaptive optimizers, and how warmup bounds the damage.
- 04 Layer Normalization and the Residual Stream How per-example statistics change the gradient path compared with batch statistics, why the placement of a normalization layer relative to a residual branch alters scale growth with depth, and what the learnable gains do.
- 05 Decoupled Weight Decay and Adaptive Scaling Why adding a shrinkage term to the gradient is not the same as shrinking the parameter, how the adaptive denominator changes the effective decay per coordinate, and what the decoupled update fixes.
- 06 Gradient Clipping by Norm and Loss-Scale Rescaling What a per-tensor global norm clip bounds, how the scale factor is computed and applied, and why clipping interacts with the loss scale used in low-precision training.
- 07 Cosine and Linear Decay Schedules as Step-Size Integrals What a learning rate schedule controls beyond the final value, how the shape distributes total parameter displacement across training, and why the choice between linear and cosine changes little at equal area.
- 08 Gradient Noise Scale and Critical Batch Size How the ratio of gradient variance to gradient squared norm sets the batch size beyond which additional examples buy no reduction in steps, and why that ratio rises over training.
- 09 Gradient Signal Propagation Through Depth at Initialization How the product of layer Jacobians determines whether a deep stack can be trained at all, why that product is easier to keep benign in residual architectures, and what the propagation analysis predicts about depth and width.
- 10 Loss-Scale Dynamics and Spike Detection in Mixed Precision Why a fixed loss scale cannot serve both the largest and smallest gradients in a run, how dynamic scaling detects overflow, and what happens to optimizer state when a step is skipped.