Tag
residual-connections
- Oct 3, 2026 Layer Normalization and the Residual Stream How per-example statistics change the gradient path compared with batch statistics, why the placement of a normalization layer relative to a residual branch alters scale growth with depth, and what the learnable gains do. deep learning
- Jan 16, 2026 Residual Connections and the Identity Gradient Path How adding a block input to its output changes the Jacobian of a stack from a product of small factors into a sum that contains the identity, what that buys in depth, and where the argument stops holding. deep learning
- May 9, 2024 Gradient Signal Propagation Through Depth at Initialization How the product of layer Jacobians determines whether a deep stack can be trained at all, why that product is easier to keep benign in residual architectures, and what the propagation analysis predicts about depth and width. deep learning