Tag
gradient-flow
- Oct 3, 2026 Layer Normalization and the Residual Stream How per-example statistics change the gradient path compared with batch statistics, why the placement of a normalization layer relative to a residual branch alters scale growth with depth, and what the learnable gains do. deep learning
- May 21, 2026 Why Scaled Dot-Product Attention Divides by the Square Root of the Key Dimension Where the 1/sqrt(d_k) factor comes from, what quantity it holds constant as width changes, and how a saturated softmax destroys the gradient reaching queries and keys. deep learning
- Jan 16, 2026 Residual Connections and the Identity Gradient Path How adding a block input to its output changes the Jacobian of a stack from a product of small factors into a sum that contains the identity, what that buys in depth, and where the argument stops holding. deep learning
- Aug 25, 2025 Backpropagation Through Time and the Product of Jacobians How gradients travel backwards through an unrolled recurrence, why the product of per-step Jacobians decays or explodes, and what truncation and clipping change about the estimate. deep learning
- May 9, 2024 Gradient Signal Propagation Through Depth at Initialization How the product of layer Jacobians determines whether a deep stack can be trained at all, why that product is easier to keep benign in residual architectures, and what the propagation analysis predicts about depth and width. deep learning
- May 8, 2024 Saturating Activations and Gradient Flow Through Depth How the derivative of a squashing function caps the signal that reaches early layers, why squaring the error makes the vanishing worst exactly when the network is most wrong, and which structural changes restore the signal. deep learning
- Jun 13, 2022 Gates in LSTM and GRU Cells: What Each Gate Controls How multiplicative gates create an additive path through time, what the forget, input and output gates each multiply, and how a two-gate cell removes one of them. deep learning