Deep Learning · Cluster
Efficient Training
Mixed precision, sharding, and memory budgets.
- 01 The Training Memory Budget: Parameters, Optimizer State, Gradients, Activations How device memory is consumed during training, why a model that fits for inference can be four times too large to train, and which term to cut first.
- 02 Choosing Between fp16 and bf16: Exponent Range and Mantissa Budget Both formats are 16 bits, but they split those bits differently. The split decides whether a training run needs loss scaling and how much precision each weight update retains.
- 03 Why a Training Step Costs Three Times Its Forward Pass The parameter-times-token accounting behind training compute budgets, where the factor of three comes from, and when attention terms make the estimate wrong.
- 04 Gradient Accumulation: What Micro-Batching Preserves and What It Does Not Splitting a batch into micro-batches keeps the arithmetic of a large batch but changes normalisation, batch-dependent layers, and the timing of every state update.
- 05 Activation Checkpointing: Trading Recomputation for Stored Tensors How discarding activations at segment boundaries and recomputing them in backward reduces memory, what the extra forward pass costs, and where the optimal segment count falls.
- 06 Ring All-Reduce and the Cost of Averaging Gradients Across Devices Why the ring algorithm moves a fixed multiple of the gradient size regardless of device count, and how bucketing hides the latency that the bandwidth term does not explain.
- 07 ZeRO Stages: Sharding Optimizer State, Gradients, and Parameters How the three partitioning stages divide training state across data-parallel ranks, what communication each one adds, and why stage 3 changes the residency model rather than just the arithmetic.
- 08 Pipeline Parallelism: Filling the Bubble and Paying for It Micro-batch pipelining lets layers be split across devices, but the schedule leaves idle slots. How bubble fraction scales with depth and why interleaved schedules help.
- 09 Blockwise 8-Bit Optimizer State Without Losing the Second Moment Quantising Adam moments to eight bits cuts state memory by four, but small second-moment values quantise to zero. Blockwise scaling and dynamic exponent sharing fix that.
- 10 Composing a Parallelism Layout From a Memory and Bandwidth Budget Data, tensor, pipeline, and shard dimensions interact non-linearly. A procedure for choosing their sizes from per-device memory, interconnect tiers, and batch constraints.