Deep Learning · Cluster
Neural Fundamentals
Layers, activations, and backpropagation.
- 01 The Fully Connected Layer as an Affine Map What a dense layer computes, why the weight matrix is transposed and the bias broadcast, and how the parameter count, the backward pass cost, and the collapse of stacked layers follow from one identity.
- 02 Pointwise Nonlinearity and the ReLU Family of Activations Why one elementwise nonlinear function is enough to break the collapse of stacked affine layers, how the rectifier behaves at its kink, and what makes a unit permanently dead.
- 03 Reverse-Mode Differentiation on the Computational Graph How a recorded graph of elementary operations turns the chain rule into one reverse sweep, why values with several consumers force an accumulation, and where that sweep silently produces wrong gradients.
- 04 Softmax Cross-Entropy and Its Collapsed Gradient Why the gradient of the cross-entropy loss with respect to logits reduces to the prediction minus the target, what that saves computationally, and how to keep the exponentials from overflowing.
- 05 Saturating Activations and Gradient Flow Through Depth How the derivative of a squashing function caps the signal that reaches early layers, why squaring the error makes the vanishing worst exactly when the network is most wrong, and which structural changes restore the signal.
- 06 Dropout Masks, Train-Time Scaling, and Inference Behaviour What the Bernoulli mask does to the expectation of an activation, why one convention scales during training instead of at test time, and how the mask changes what reaches the layers below.
- 07 Residual Connections and the Identity Gradient Path How adding a block input to its output changes the Jacobian of a stack from a product of small factors into a sum that contains the identity, what that buys in depth, and where the argument stops holding.
- 08 Jacobian-Vector and Vector-Jacobian Products in Autodiff Why differentiation machinery never materializes a Jacobian, the cost rule that decides whether to sweep forward or backward, and how the same primitives produce exact Hessian-vector products.
- 09 Gradient Checking by Finite Differences The truncation and round-off errors that bound how accurately a numerical derivative can be measured, how to choose the step size and the comparison metric, and why a check can pass on a broken derivative.
- 10 Depth, Width, and the Linear Regions of ReLU Networks Every rectified network is a partition of its input space into cells where it computes an affine map, and the count of those cells grows polynomially in width and exponentially in depth.