Saturating Activations and Gradient Flow Through Depth

The logistic function was the standard unit of early feedforward networks, and it is still the natural unit wherever an output is a probability or a gate. Used as the activation of a deep stack it has a property that is easy to state and hard to overstate: its derivative is never larger than $0.25$, and it decays exponentially as the pre-activation grows in magnitude. Since the reverse sweep multiplies one such factor per layer, a chain of logistic units attenuates the gradient geometrically with depth. The question is not whether the signal shrinks but how the shrinkage compares with everything else in the product.

The derivative ceiling of a squashing function

For $\sigma(z) = 1/(1 + e^{-z})$ the derivative is $\sigma(z)(1 - \sigma(z))$, a parabola in the output with its maximum at $\sigma = 0.5$, where it equals exactly $0.25$. For large positive $z$ the derivative behaves like $e^{-z}$, and for large negative $z$ like $e^{z}$; a unit whose pre-activation is 8 has a derivative near $3 \times 10^{-4}$, so a single layer can attenuate by more than three orders of magnitude without any depth involved. The hyperbolic tangent has the same shape shifted around zero: $\tanh’(z) = 1 - \tanh^2(z)$, maximized at 1 when $z = 0$, decaying like $4e^{-2|z|}$ in the tails. Its ceiling is four times higher than the logistic ceiling, which is one of the standard arguments for a symmetric, zero-centred activation.

Since the reverse sweep multiplies the local derivative of each layer into the cotangent, the magnitude reaching the first layer is bounded by the product of the per-layer factors. Ten logistic units in a chain, each at its maximum derivative, give $0.25^{10} \approx 9.5 \times 10^{-7}$; at a pre-activation of 4 per layer the factor is closer to $0.018$ and the ten-layer product is about $10^{-17}$. Floating-point arithmetic in 32-bit precision has roughly seven decimal digits of relative precision, so at that point the gradient reaching the early layers is not merely small, it is indistinguishable from rounding error. In 64-bit arithmetic the same product survives, which is one reason the problem appears with depth rather than at a fixed magnitude.

The shrinking gradient also has a second-order effect on the direction of the step rather than only its size. Because the logistic output lies in $(0,1)$ and never takes negative values, the inputs to the next layer are all positive, and the gradient with respect to that layer’s weights is an outer product of a positive vector with the incoming cotangent. All its entries then share the sign pattern of the cotangent, so the update direction moves along a restricted set of directions instead of the full space. This is the argument made for centring the activation so that its output is roughly symmetric around zero; it is a property of the shape of the activation’s range, not of its derivative.

Why squaring the error makes the failure worse than it looks

The magnitude argument above assumes the local derivative is merely small. Pairing a logistic output unit with a squared-error loss makes the gradient of the loss with respect to the unit’s weights contain that derivative as a factor multiplying the error, so the two effects multiply instead of one substituting for the other. If the target is 1 and the output is 0.001, the error term is $0.999$ but the derivative factor is $\sigma’(z) = 0.001 \cdot 0.999 \approx 1 \times 10^{-3}$, and their product is about $10^{-6}$. The unit is maximally wrong and receives almost no signal. Cross-entropy on the same unit does not contain the factor at all: its derivative with respect to the logit is $a - y$, which is $-0.999$ in exactly that case. The cancellation that produces this is the binary case of the identity derived in softmax cross-entropy, and it is the reason a logistic output unit is almost always paired with the log loss rather than with squared error.

Squared error is not wrong in general; it is wrong when its error term is applied to a squashed output. For a linear output unit predicting a real value, $a - y$ has no derivative factor, and the same squared error gives a well-scaled gradient at any magnitude of error. The pairing, not the loss, is the design decision.

Saturation is not a defect in every context. A multiplicative gate is a squashing function whose purpose is to move between open and closed, and the flat tails are what make the closed state stable: a gate whose pre-activation is deep in the tail changes little when its inputs change, so the state it protects does not drift with every perturbation of the values feeding the gate. The price is the same one identified above in the other direction. A gate saturated closed receives a derivative near zero, so reopening it requires many steps in which it is already nearly open, and an initialization that leaves a gate closed can prevent the state from ever being used. Initializing the gate bias so that the gate starts open is the standard remedy in recurrent architectures that gate a state, and it is the mirror image of the dead-unit problem in a rectified stack: both are cases of a unit sitting where its own derivative is zero.

Structural responses to an attenuated product

Three responses are visible in architectures that are otherwise quite different.

The first is to choose an activation whose derivative is not bounded below 1 on the region that matters. The rectifier has a derivative of exactly 1 for every positive pre-activation, so the product across depth decays only where units are inactive, as described in activation functions. This removes the inherent ceiling rather than compensating for it.

The second is to give the gradient a path that does not pass through any activation. An addition of the block input to the block output contributes an identity term to the block’s Jacobian, so the product over layers is no longer a product of small factors alone; the mechanism is treated in residual connections. Gating is the related idea in the other direction: a multiplicative gate that is itself a sigmoid controls how much of a state passes through, and the derivative of that gate is the factor that decides whether the state is preserved or overwritten.

The third is to control the scale of the pre-activations themselves, since saturation is what drives the derivative into the tail. Normalization layers keep the pre-activations in a range where the derivative is not negligible, and initialization schemes set the weight scale so that the variance of the pre-activation is approximately preserved across layers rather than growing or shrinking with depth. The interaction between the two matters: a normalization placed inside a residual branch rescales the branch, not the identity path, so the branch and the skip path end up with different scales.

Reading the symptom

A gradient that vanishes and a model that cannot fit a small dataset look similar and have different causes. The distinguishing observation for the vanishing case is that the magnitude of the gradient with respect to the earliest layers’ parameters is many orders of magnitude below that of the last layer’s, measured directly on the same batch. The derivative ceiling bounds the ratio between consecutive layers, so a depth of $L$ with a mean per-layer factor $f$ implies a ratio of roughly $f^{L}$ between the first and last layer. When the activations are saturated the factor is far below the ceiling: a unit whose pre-activations are large has a derivative that decays exponentially in their magnitude, so the ratio between layers can exceed the ceiling’s exponent by a large factor.

The reverse pathology, in which the product of per-layer factors exceeds 1 and the gradient grows with depth, has the same arithmetic and different arithmetic consequences: an exponentially growing cotangent overflows the exponent range of 32-bit floats, at which point the parameters become non-finite and every subsequent step propagates the non-finite value. Both directions are properties of the product of Jacobians along the path, and neither is specific to a particular activation; a squashing activation merely places the product’s ceiling below 1 by construction.