Tag
numerical-stability
- Apr 18, 2025 Catastrophic Cancellation and the Cases Where Subtraction Is Exact Why subtracting two nearly equal numbers amplifies existing error rather than creating it, when the difference is exact by Sterbenz, and how formulas are rewritten to avoid the loss. scientific computing
- Feb 11, 2025 Gradient Clipping by Norm and Loss-Scale Rescaling What a per-tensor global norm clip bounds, how the scale factor is computed and applied, and why clipping interacts with the loss scale used in low-precision training. deep learning
- Sep 29, 2024 Solving Quadratic Ray Intersections Without Losing the Far Root Why the textbook quadratic formula fails for distant spheres, how the stable form differs by an addition of magnitudes, and what the determinant sign means geometrically. computer graphics
- Jul 4, 2024 Loss-Scale Dynamics and Spike Detection in Mixed Precision Why a fixed loss scale cannot serve both the largest and smallest gradients in a run, how dynamic scaling detects overflow, and what happens to optimizer state when a step is skipped. deep learning
- Apr 12, 2022 Softmax Cross-Entropy and Its Collapsed Gradient Why the gradient of the cross-entropy loss with respect to logits reduces to the prediction minus the target, what that saves computationally, and how to keep the exponentials from overflowing. deep learning