Tag
activation-functions
- Aug 14, 2025 Variance Preservation in Weight Initialization How the Xavier and He scaling factors are derived from the fan-in of a layer, what breaks when the constant is wrong, and why depth turns a small gain error into a vanished signal. deep learning
- Mar 16, 2025 What a Recurrent Cell Computes at Each Time Step The exact arithmetic of an RNN cell, the tensor shapes and cost per step, why the recurrence is unrolled for training, and how hidden states behave when the sequence continues across batches. deep learning
- May 8, 2024 Saturating Activations and Gradient Flow Through Depth How the derivative of a squashing function caps the signal that reaches early layers, why squaring the error makes the vanishing worst exactly when the network is most wrong, and which structural changes restore the signal. deep learning
- Aug 25, 2023 Pointwise Nonlinearity and the ReLU Family of Activations Why one elementwise nonlinear function is enough to break the collapse of stacked affine layers, how the rectifier behaves at its kink, and what makes a unit permanently dead. deep learning