Tag
mixed-precision
- Oct 16, 2026 Choosing Between fp16 and bf16: Exponent Range and Mantissa Budget Both formats are 16 bits, but they split those bits differently. The split decides whether a training run needs loss scaling and how much precision each weight update retains. deep learning
- May 3, 2025 Layer Sensitivity and Bit Allocation Under an Average-Width Budget How per-layer sensitivity is measured, why a greedy allocation on a non-additive error metric misleads, and how to spend a fixed average bit width across a network. inference systems
- Feb 11, 2025 Gradient Clipping by Norm and Loss-Scale Rescaling What a per-tensor global norm clip bounds, how the scale factor is computed and applied, and why clipping interacts with the loss scale used in low-precision training. deep learning
- Sep 13, 2024 The Training Memory Budget: Parameters, Optimizer State, Gradients, Activations How device memory is consumed during training, why a model that fits for inference can be four times too large to train, and which term to cut first. deep learning
- Jul 4, 2024 Loss-Scale Dynamics and Spike Detection in Mixed Precision Why a fixed loss scale cannot serve both the largest and smallest gradients in a run, how dynamic scaling detects overflow, and what happens to optimizer state when a step is skipped. deep learning
- Dec 11, 2023 Activation Outlier Channels and Mixed-Precision Decomposition Why a few hidden-state channels carry magnitudes far above the rest, how splitting those channels out of the integer path preserves accuracy, and what the split costs per token. inference systems