Tag
signal-scaling
- Oct 3, 2026 Layer Normalization and the Residual Stream How per-example statistics change the gradient path compared with batch statistics, why the placement of a normalization layer relative to a residual branch alters scale growth with depth, and what the learnable gains do. deep learning
- Aug 14, 2025 Variance Preservation in Weight Initialization How the Xavier and He scaling factors are derived from the fan-in of a layer, what breaks when the constant is wrong, and why depth turns a small gain error into a vanished signal. deep learning