Pointwise Nonlinearity and the ReLU Family of Activations

Two stacked affine layers are one affine layer, so depth is meaningless without something in between. The smallest such thing that works is a scalar function applied to each element of the layer output independently. That restriction is severe, and it is almost always what is used, because an elementwise function has an elementwise Jacobian: the backward pass through it is a multiply by a vector of local derivatives, with no mixing between units and no coupling between samples in a batch.

Why elementwise is enough

Composing an affine map with a per-coordinate function produces a function that is not affine, and composing more of them produces something larger still. In one dimension, $f(x) = \mathrm{relu}(w_2,\mathrm{relu}(w_1 x + b_1) + b_2)$ is piecewise linear with two breakpoints, and each additional rectified unit in the chain can add breakpoints that the previous unit’s structure did not place there. The pointwise restriction is what keeps the Jacobian cheap: for $a = \phi(z)$ with $\phi$ applied coordinatewise,

$$\frac{\partial a_i}{\partial z_j} = \begin{cases} \phi’(z_i) & i = j \ 0 & i \neq j \end{cases}$$

so the vector-Jacobian product needed by the backward pass is an elementwise product of the incoming cotangent with $\phi’(z)$. This is a memory-bandwidth-bound operation over a tensor that already exists, not a matrix product, and it costs one pass over the elements rather than one per pair of them. It also preserves independence across the batch axis, unlike a normalization layer or any operation that computes statistics over samples.

The cost of an elementwise activation is set by memory traffic rather than by arithmetic. A pass over an activation tensor holding $N \times k$ values in 32-bit floats reads $4Nk$ bytes and writes the same again, while performing one comparison and one multiply per element, so the operation is bandwidth-bound by a wide margin. What it does not contain is a reduction: no sum over the batch, no maximum over channels, no dependence of one element’s result on another’s. That matters for the surrounding machinery, because a reduction forces a synchronization point between the values being combined, whereas an elementwise map parallelizes across every axis including the batch axis. The exception is a mask whose value depends on a statistic computed elsewhere in the tensor, which is what makes an operation that normalizes over samples different in kind rather than merely different in cost.

The rectifier: constant slope, one kink, and an exact zero region

The rectified linear unit is $\mathrm{relu}(z) = \max(0, z)$. Its derivative is 1 for $z > 0$ and 0 for $z < 0$, and it is undefined at exactly zero, where implementations conventionally return 0 rather than 1. Two properties follow from that derivative being constant rather than bounded by a curve. First, a unit with a large positive pre-activation propagates the full incoming gradient: there is no saturation, so increasing magnitude does not attenuate the signal. This is the opposite of what a logistic unit does at the same magnitude, as described in saturating activations. Second, the second derivative is zero almost everywhere, so methods that need curvature information get none from the activation itself.

The zero region of the derivative is the source of the rectifier’s characteristic failure. For a fixed input, if $z_i \le 0$ for every sample in the batch, then the unit’s output is zero, the incoming cotangent is multiplied by 0, and the unit’s weight and bias gradients are exactly zero. Its parameters do not move in that step, and since the parameters determine whether the pre-activation is negative in the first place, they tend not to move in later steps either. A unit in that state contributes nothing to the function and receives nothing back from it. Large gradient steps on a wide layer, or an initialization that produces pre-activations with large negative mean, are the usual ways to arrive there.

Leaky variants change the negative slope from 0 to a small positive constant, commonly 0.01 for the leaky rectifier, which means the negative side transmits one percent of the incoming gradient. That is small, but it is not zero, so the unit’s weights keep receiving a signal and the unit can climb back out of the region. A popular parameterization of the same idea replaces the fixed slope with $\exp(z) - 1$ on the negative side and $z$ on the positive side, which is continuous and has a continuous derivative at zero, removing the kink at the cost of a transcendental function evaluation per element.

import numpy as np

def relu(z):
    return np.maximum(0.0, z)

def relu_backward(dA, z):
    # the mask is recomputed from the pre-activation, not from the output
    return dA * (z > 0.0)          # zero at exactly z == 0

def leaky_relu(z, slope=0.01):
    return np.where(z > 0.0, z, slope * z)

def leaky_relu_backward(dA, z, slope=0.01):
    return dA * np.where(z > 0.0, 1.0, slope)

# A unit dead for every sample in the batch has no gradient at all.
z = np.array([[-2.0, 0.5], [-3.0, 1.5]])
assert np.all(relu_backward(np.ones_like(z), z)[:, 0] == 0.0)

Smooth and saturating relatives

The softplus function $\log(1 + e^{z})$ has derivative $\sigma(z)$, the logistic function, so its slope goes to 0 as $z \to -\infty$ and to 1 as $z \to +\infty$ without ever being exactly zero. A unit using it is never exactly dead, but for pre-activations around $-20$ the derivative in 32-bit arithmetic is small enough that the unit is effectively frozen. The Gaussian error linear unit, $z,\Phi(z)$ where $\Phi$ is the standard normal cumulative distribution, is smooth and non-monotone: it has a shallow negative dip for moderate negative inputs before rising toward the negative asymptote. Its commonly used hyperbolic tangent form is

$$\mathrm{gelu}(z) \approx 0.5z\left(1 + \tanh\left(\sqrt{2/\pi},\left(z + 0.044715 z^3\right)\right)\right)$$

which is an approximation, not the exact definition, and differs from it in the tails. Both functions cost more per element than a max, and neither has a derivative that is exactly zero over an interval.

The logistic and hyperbolic tangent functions remain in use where a bounded output is wanted. Their derivatives satisfy $\sigma’(z) = \sigma(z)(1-\sigma(z)) \le 0.25$ and $\tanh’(z) = 1 - \tanh^2(z) \le 1$, with both ratios decaying exponentially in $|z|$. Bounded output and bounded derivative are the same property seen twice, and it is why these functions are the natural choice for gates, where an output in $(0,1)$ is the semantic content, and a poor choice for stacking dozens of layers deep.

Because the cost of an activation is dominated by memory traffic rather than by arithmetic, replacing a rectifier with a smoother unit changes the runtime by a modest factor and changes the gradient behaviour substantially; this is not a throughput decision. The derivative is what differs. The rectifier’s is exactly 0 or exactly 1 over intervals, so a unit is either fully live or fully dead for a given input. A smooth unit’s derivative lies strictly inside $(0,1)$, so no unit is ever exactly dead, and some are effectively so: at a pre-activation of $-20$ the logistic derivative is about $2\times10^{-9}$, which is below the resolution that a 32-bit cotangent can carry once it is multiplied into typical downstream factors.

Where the activation is placed and what it can express

The activation sits between the affine map and whatever consumes the layer’s output, and its position relative to other operations is a design decision rather than a formality. Placing it on the layer output means the value stored for the backward pass is the activation, and the derivative must be recomputed from the pre-activation or recovered from the output. Placing it before the next affine map, with a normalization or a residual addition in between, changes what the stored tensors mean and therefore which quantity the backward pass multiplies, a distinction that matters once blocks are composed as in residual connections.

At the output of the network the choice is dictated by the range of the target. A nonlinearity with a bounded range caps the function the network can represent, which is acceptable for a probability and wrong for a real-valued regression target. A rectifier at the output imposes a floor at zero, and it does so with a derivative of exactly zero on the negative side, so an output unit that starts on the wrong side of that kink gets no signal at all. Multiclass outputs use the exponential normalizing function, whose gradient calculation is treated in softmax cross-entropy.

Any stack whose activations are piecewise linear, including every rectifier variant, produces a function that is itself piecewise linear: the input space is cut into regions by the kinks, and inside each region the network computes an affine map. The number of such regions is the subject of linear region counting, and it is the reason depth in particular is worth paying for.