What a Recurrent Cell Computes at Each Time Step

A recurrent layer exposes an interface that hides everything interesting about it. The caller supplies a sequence of shape $T \times n$ and receives an output sequence of shape $T \times d$, and nothing in the call signature indicates that this is $T$ separate applications of one small function whose output feeds its own input. Treating the layer as a black box is usually fine, until sequence lengths vary, states must be carried across batches, or the loss diverges after a few hundred steps. All three of those situations are explained by the step-level arithmetic.

The update in full

One step is an affine map of the concatenated current input and previous state, followed by a squashing function:

$$h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b_h)$$

With input width $n$ and state width $d$, the two weight matrices have shapes $d \times n$ and $d \times d$, and the bias has $d$ entries, giving $d(n + d) + d$ parameters for the cell regardless of sequence length. Folding the two matrices into one of shape $d \times (n+d)$ and the input and state into one vector is equivalent arithmetic and is what an optimized implementation does, because a single matrix-vector product uses the machine better than two. The cost per step is on the order of $d(n + d)$ multiply-accumulates, so a full sequence costs $O(T \cdot d(n+d))$ time and holds $O(T \cdot d)$ activations when gradients are needed for every step.

The state width $d$ is the entire carrying capacity of the model. A cell with $d = 128$ holds 128 numbers; the map from prefix to those numbers is what training shapes. There is no separate memory and no growth in state size with sequence length, which is what makes the layer’s memory use predictable and also what limits it.

Initialization of $W_{hh}$ controls behaviour before training has anything to say. If the spectral radius of $W_{hh}$ exceeds one, repeated multiplication by the same matrix amplifies the state along its dominant direction, and the value grows geometrically; below one it contracts and earlier inputs are washed out. Orthogonal initialization of $W_{hh}$ sets all singular values to one so that the norm of the state is preserved in the linear part, which keeps the signal from exploding or vanishing before the nonlinearity is applied. That is a statement about the linear map, not about the trained model.

Unrolling as a computational graph

Training treats the recurrence as a deep network with $T$ layers that happen to share weights. The distinction matters for two reasons. First, weight sharing means the gradient contributions from every step accumulate into one set of parameters, so a single bad step contributes to the same matrix as every good step. Second, the graph must be retained: either the framework stores per-step activations, at $O(T \cdot d)$ values, or the user supplies a custom backward pass that recomputes them.

The sequential structure also determines latency. Total arithmetic is linear in $T$, but the critical path is $T$ dependent matrix products, so a step cannot be vectorized across time the way a convolution is vectorized across positions. Batching increases utilization without shortening the chain, because the batch dimension is independent of the recurrence. This is the same axis described in fixed-window models and hidden state: a feedforward window has a critical path of one layer and a receptive field fixed in advance.

Choosing the nonlinearity

$\tanh$ was the original choice and remains the common default. It is bounded, so the state cannot grow without limit through the nonlinearity alone, and it is zero-centred, which keeps the affine input to the next step roughly balanced. Its derivative is $1 - \tanh^2$, which is at most one and falls to zero in saturation, so gradients passed back through the activation shrink at every step. A rectified linear unit has derivative exactly one for positive pre-activations, which avoids that particular shrinkage, but it is unbounded and permits the state to drift upward; a network of ReLUs applied to its own output tends toward large positive values unless the weights are small. Neither choice removes the fundamental decay described in backpropagation through time, since the Jacobian of the state-to-state map is $W_{hh}^{\top}$ multiplied by the derivative of the activation, and its spectral properties are what govern the long-range gradient.

Initial state and continuity across batches

The state at the first step of a sequence has to come from somewhere. Zero is the usual convention, and it defines the cell’s output at the first positions as a function of the first input and a constant. Training on documents then teaches the model that a document starts from a blank state, which is a false assumption when inference runs over a stream cut into chunks. Two remedies exist. The first is a learned initial state: a parameter vector of width $d$ used at $t = 0$ for every sequence, which lets one state serve as the accumulator for information that is genuinely present before the window begins. The second is passing the final state of one chunk as the initial state of the next, which preserves continuity but couples the chunks and complicates batching, since sequences in a batch must then be ordered.

The continuity pattern changes what the model sees. With zero initialization on overlapping windows the model never observes a state that was produced by its own recurrence over a longer span, so it learns a distribution over initial conditions that inference violates. Detaching the carried state from the graph between chunks bounds memory while keeping the values, which is the common arrangement for streaming inference over long inputs.

Saturation and the resolution of the state

The bounded activation that keeps the state finite also bounds what the state can represent. In 32-bit floating point, $\tanh$ returns exactly $1.0$ for arguments beyond roughly $9$, because $1 - \tanh^2 x$ falls below the spacing of representable values near one; the backward pass multiplies by that derivative and therefore contributes exactly nothing at those coordinates. A cell whose pre-activations drift into that region stops differentiating with respect to its own weights and, if the drift is persistent, stops moving. The same effect appears on a smaller scale long before the value pins at one: the derivative shrinks quadratically with distance from the origin, so coordinates that sit far from zero transmit proportionally less gradient. States in a trained model cluster near zero partly because that is where the activation is most responsive.

The opposite failure is bounded by the format rather than the function. Repeated multiplication by a matrix with singular values above one grows the state geometrically, and a 16-bit float cannot represent magnitudes past a few tens of thousands, so a state that grows by a few percent per step leaves the representable range within a few thousand steps. Mixed-precision training of recurrent layers therefore keeps the recurrence itself in 32-bit arithmetic in the usual arrangement, or clips the state.

Detecting a recurrence that is not training

Three quantities make most recurrent training failures visible. The first is the norm of the state at each step, which reveals geometric growth or collapse over a sequence before the loss becomes unusable. The second is the norm of the gradient with respect to the recurrent weights, compared against the norm with respect to the input weights; when the recurrent one is orders of magnitude smaller, the long-range paths are contributing nothing. The third is the sensitivity of the output at the last position to a perturbation of the input at the first, which measures the reach of the trained state directly.

Rescaling the gradient when its global norm exceeds a threshold is the standard remedy for the growth case. Clipping changes the direction of an update only in the sense of shortening it, and it leaves the forward state untouched. The decay case has no equivalent fix at training time, because the gradient that would carry the signal out of the early steps is already numerically zero; it is addressed by changing the recurrence itself, as in gating, or by shortening the paths the signal has to travel through truncation.

Readout and loss placement

The cell produces a state; a separate projection maps it to whatever the task needs. For a per-step classification the same matrix of shape $|V| \times d$ is applied at every position, and the loss averages over the $T$ positions, so each step contributes gradient to the shared weights. For a single label per sequence the projection is applied only to the last state, and the earlier steps receive gradient only through the recurrence. The second arrangement wastes most of the label information: a sequence of 500 steps with one label at the end produces a gradient path of length 500 back to the first input, which is the longest possible chain for that information to travel.

import numpy as np

def step(x, h, Wxh, Whh, bh):
    # one recurrence step; x is (n,), h is (d,)
    return np.tanh(Wxh @ x + Whh @ h + bh)

def unroll(xs, Wxh, Whh, bh, h0=None):
    d = Whh.shape[0]
    h = np.zeros(d) if h0 is None else h0
    states = []
    for x in xs:              # sequential: each step needs the previous state
        h = step(x, h, Wxh, Whh, bh)
        states.append(h)
    return np.stack(states)

The loop above is the whole layer. Every property that makes recurrent models awkward to train and to serve follows from the fact that the loop cannot be reordered, and from the fact that the same weights are differentiated with respect to at every iteration.