Backpropagation Through Time and the Product of Jacobians

Backpropagation through time is reverse-mode differentiation applied to an unrolled recurrence, and it is not a special algorithm with its own rules. The recurrence is a graph with $T$ nodes that share weights, and the chain rule is applied to it exactly as to a feedforward stack. What makes the case worth examining separately is the object that appears in the middle of the calculation: the gradient from a late loss to an early state is carried by a product of $T - t$ matrices, and the behaviour of that product decides whether the early steps are trained at all.

The gradient written out

For a loss summed over positions, $\mathcal{L} = \sum_t \mathcal{L}t$, the derivative with respect to the recurrent matrix is itself a sum of per-position terms. Each term expands into a sum over the step $j$ at which the direct dependence on $W{hh}$ occurs:

$$\frac{\partial \mathcal{L}}{\partial W_{hh}} = \sum_t \sum_{j \le t} \left(\frac{\partial \mathcal{L}t}{\partial h_t}\right) \left(\prod{k=j+1}^{t} \frac{\partial h_k}{\partial h_{k-1}}\right) \frac{\partial h_j}{\partial W_{hh}}$$

Each factor in the inner product is the Jacobian of one recurrence step, $\partial h_k / \partial h_{k-1} = \mathrm{diag}(\phi’(z_k)), W_{hh}^{\top}$, a $d \times d$ matrix, so a dependency that spans $\Delta t$ steps is scaled by a product of $\Delta t$ matrices rather than by one. The sum over $j$ means the total gradient mixes contributions of many different lengths, and the long ones are exactly the ones that shrink.

A scalar approximation makes the shape of the problem concrete. If the per-step factor has norm $\rho$, the influence of an early step decays or grows like $\rho^{\Delta t}$. With $\rho = 0.9$, a dependency spanning 50 steps is scaled by roughly $0.005$; with $\rho = 1.1$, the same span is scaled by roughly $117$. Both directions are failures, and the interval in which $\rho$ must sit for thousands of steps to matter at all is narrow by construction.

Two practical consequences follow from the sum structure. The first is that the size of the gradient depends on how many loss terms are summed. A dense per-step loss contributes $T$ terms whose short-range parts are of similar magnitude, so the total grows with $T$ and a learning rate tuned on short sequences overshoots on long ones unless the loss is averaged over positions. The second is that a parameter receives gradient from steps it influenced only indirectly, so one position with an unusually large contribution can dominate the sum for that parameter, which is how a few atypical sequences end up steering the update.

Why the per-step factor usually sits below one

Two multipliers drive $\rho$ down. The activation derivative $\phi’$ is at most one for $\tanh$ and equals one only at zero, so any step whose pre-activation is away from the origin contributes a contraction. The matrix $W_{hh}^{\top}$ contributes its singular values, and after the usual initialization schemes they are spread around values below one for the state widths in common use. The product of many such factors behaves like the product of many random matrices, and its norm typically decays exponentially in the number of factors with a rate set by the average log-singular-value along the dominant direction. The result is that a gradient signal from step $t$ arrives at step $j$ attenuated by an amount that is exponential in $t - j$.

Hochreiter’s analysis in the early 1990s identified this as the central obstacle to learning dependencies that span long distances, and the practical consequence is visible in the training dynamics rather than in the loss. A model can reach a low loss while the gradient with respect to the recurrent weights is negligible, because the short-range paths that dominate the sum are sufficient for the objective. The early positions then stop contributing and the state’s long-range content is never learned, which is indistinguishable from an architecture that has no long-range capacity.

Note that dilution is not the only failure. A gradient signal can be entirely adequate and still be unhelpful if the objective offers no evidence that long-range information should be retained. The vanishing gradient is a statement about the path, not about the value of the path.

Truncation as a deliberate bias

The unrolled graph has one node per step, so a forward and backward pass over a sequence of length $T$ retains $O(T \cdot d)$ activation values. Truncated backpropagation through time replaces this with a sequence of segments: the forward pass runs over a segment of $C$ steps carrying the state, the backward pass runs over the same segment, and the carried state is detached from the graph at the boundary so no gradient crosses it. Peak memory becomes $O(C \cdot d)$ and the backward pass cost becomes $O(C \cdot d^{2})$ per segment, so both become independent of the total length.

The price is bias. Parameters receive the exact gradient contribution of dependencies no longer than $C$ steps, and no contribution at all from longer ones. Segments of a few tens of steps were the original proposal, and lengths from a few tens to a few hundred remain the practical range, chosen to balance memory against the longest dependency the task actually contains. Two details matter for correctness: the state values must still be propagated across segment boundaries even though the gradient is not, and any normalization or loss computed per segment must not be allowed to depend on the segment split, or training and inference disagree.

A related arrangement processes the whole sequence in one graph but detaches the state at every step except the current one, which yields a purely local update. It removes the product entirely, so it cannot learn any long-range dependency, and it is used where the recurrence acts as a fixed feature extractor rather than as a learned memory.

Exploding gradients and norm clipping

When $\rho$ exceeds one, the same product that attenuates long-range contributions amplifies short-range ones, and a single unlucky step can produce a parameter update large enough to undo many steps of progress. The value that is actually exploding is not always the gradient: the forward state itself grows geometrically under repeated multiplication by the same matrix, and once it saturates the squashing function the model’s response to input disappears even though the loss looks finite.

The standard remedy is to compute the norm of the gradient over all parameters, and if it exceeds a threshold, rescale the whole gradient vector to that threshold before the optimizer sees it. The direction is preserved; only the length is bounded. This is a change to the update, not to the gradient computation, and it does nothing for the decaying case, since scaling a vector that is numerically zero leaves it zero. A clipped threshold also fixes the effective learning rate for the extreme steps, which is why it interacts with schedules designed around a nominal rate.

import numpy as np

def bptt_scalar(a, xs):
    """dL/dw for L = 0.5*h_T**2 with h_t = a*h_{t-1} + w*x_t, evaluated at w = 1."""
    h = [0.0]
    for t in range(len(xs)):
        h.append(a * h[-1] + xs[t])
    dh = h[-1]                    # dL/dh_T
    grad = 0.0
    for t in range(len(xs), 0, -1):
        grad += dh * xs[t - 1]    # direct dependence on w at this step
        dh *= a                   # multiply by the single-step Jacobian
    return grad, a ** len(xs)     # second value: weight of the oldest step alone

Running this with $a = 0.9$ over fifty steps prints a total gradient dominated by the last few steps while the oldest step’s contribution is a factor of $0.9^{50}$ smaller. The reverse loop above is the whole of the algorithm; the framework versions differ only in carrying a matrix instead of a scalar and in storing the activations needed to recompute derivatives.

Conditioning the recurrence instead

Three structural changes move the per-step factor toward one. Orthogonal or unitary parameterization constrains the recurrent matrix so that all its singular values equal one, which leaves the activation derivative as the only source of decay; the price is an extra cost per step to maintain the constraint, and the nonlinearity still shrinks the signal in saturated regions. Gating introduces a path through the state whose per-step multiplier is a learned scalar rather than a full matrix, so the error can be held without passing through a matrix product at all, which is the mechanism examined in gates in LSTM and GRU cells. Adding the previous state to the cell’s output puts an identity in the Jacobian, $I + J$, whose eigenvalues cluster near one and therefore neither vanish nor blow up, an idea that reappears in the depth direction and is described in bidirectional and stacked recurrence.

Each of these trades something: orthogonality costs compute, gating costs parameters and three or four matrix products per step, and residual addition changes what the cell’s output means. None of them removes the need to choose where the graph is cut, because the memory of an unrolled graph still grows with the span over which gradients are kept. Truncation remains the mechanism that makes the training cost independent of sequence length, and its bias remains the reason long-range dependencies need either a longer segment or a recurrence whose Jacobian is close to norm-preserving.