Gates in LSTM and GRU Cells: What Each Gate Controls

The plain recurrent cell of the previous step overwrites its state. Every update writes a new value into every coordinate, and the only thing that survives from the previous state is whatever the matrix multiplication preserves. A cell that must hold a count, an open constituent, or a decision made a hundred steps earlier has no way to protect that information from being overwritten by the next input. Gating is the mechanism that turns unconditional overwriting into conditional overwriting, and the LSTM cell is best read as a consequence of that single change.

An additive path with a learned multiplier

The long short-term memory cell separates the state from the output. It maintains a cell state $c_t$ that is updated additively rather than by replacement:

$$c_t = f_t \odot c_{t-1} + i_t \odot g_t$$

where $f_t$ is the forget gate, $i_t$ the input gate, and $g_t$ a candidate value computed from the input and the previous output state, all of width $d$ and all produced by affine maps passed through logistic squashing for the gates and $\tanh$ for the candidate. The visible output state is a filtered view of the cell state, $h_t = o_t \odot \tanh(c_t)$, with $o_t$ the output gate.

The whole point is in the derivative of that first equation with respect to the previous cell state: $\partial c_t / \partial c_{t-1} = \mathrm{diag}(f_t)$. There is no matrix product. Gradient travelling backwards along the cell state is multiplied at each step by the forget gate value, so if the gate sits near one the gradient passes through unchanged, and if it sits near zero the path is closed. Compare that with the Jacobian of the plain cell, $\mathrm{diag}(\phi’), W_{hh}^{\top}$, whose product over many steps is the source of the decay described in backpropagation through time. The forget gate makes the per-step multiplier a learned scalar, and a gate driven by the input can hold that scalar near one for exactly as long as the information is needed.

The bound still exists. A forget gate that averages $0.99$ attenuates a signal by a factor of about $0.37$ over 100 steps and about $4\times10^{-5}$ over 1000 steps, so nothing is preserved indefinitely by a gate that is merely close to one; what changes is that the decay rate is a parameter of the model rather than an artifact of a random matrix spectrum.

All three gates are the same computation

Every gate is an affine map of the concatenation of the current input and the previous output state, followed by a logistic squash, so the cell computes four such maps per step. With input width $n$ and state width $d$ the four weight matrices total $4d(n + d) + 4d$ parameters, which is four times the parameter count of the plain cell. The difference in meaning lies in what each result multiplies.

The input gate $i_t$ scales the candidate before it is added, so it decides whether this step writes at all. The forget gate $f_t$ scales the previous cell state, so it decides how much history survives the write. The output gate $o_t$ scales the exposed state, which decouples what is stored from what is visible to the rest of the network and to the next step’s gates, since those depend on $h_{t-1}$ and not on $c_{t-1}$.

The reason a cell needs both a write control and an erase control is that they answer different questions, and tying them together forces an all-or-nothing write. A tied cell either keeps everything and writes nothing or writes and forgets everything at the same step. The original forget gate formulation omitted the erase control entirely, which works only for sequences whose state is reset by the data itself, that is, tasks where the segment boundary is known and the cell can be reinitialized there. Adding the forget gate makes the reset a learned event, and initializing its bias to a positive value so the gate starts open avoids the situation in which the gate begins near zero and the gradient along the cell state is blocked from the first step.

A two-gate cell with the same idea

The gated recurrent unit keeps the additive interpolation but drops the separation between stored and exposed state. It computes an update gate $z_t$ and a reset gate $r_t$, both of width $d$, and produces

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$

where the candidate $\tilde{h}_t$ is computed from the current input and the previous state after the reset gate has had its say. The interpolation form means $z_t = 0$ copies the previous state exactly and $z_t = 1$ replaces it, so the same gate plays both the keep and write roles, and the two roles cannot be set independently. In exchange the cell computes three affine maps per step instead of four, so it has three quarters the parameters and three quarters the multiply-accumulates of an LSTM of the same width, and the exposed state is the state, which removes one $\tanh$ and one elementwise product per step.

Which of the two is preferable is not settled by theory. The extra degree of freedom in the LSTM makes it strictly more expressive in the sense that its state can be held while the output is suppressed, and the cost is a quarter more arithmetic per step and per parameter. The comparison that matters in practice is at equal per-step compute, where the two cells trade width against gate count.

Reading gate values

The gates have a reading that makes the cell inspectable. A forget gate near one at a given coordinate means the cell state is being carried forward; near zero it means that coordinate is being cleared, and a model that clears the same coordinate at every step has effectively reduced its state width. The input gate and candidate together decide the increment that is written, and the output gate decides how much of the stored value the rest of the network observes, so a coordinate can hold a value for many steps while suppressing it and expose it only when the output gate opens. Those readings are properties of the mechanism rather than of any particular trained model.

The gate biases decide the behaviour of the cell before training has shaped it, which is why they are set deliberately. A forget gate with a strongly negative bias starts closed, so the cell state stays near zero at every step and the gradient along it is blocked from the outset; a bias near zero or slightly positive starts the gate open, letting the error signal reach the early steps long enough for the gate to learn when to close. A closed input gate is a milder failure, since it only prevents new content from being written while leaving what is already held intact.

What gating does not fix

A gate is a function of the previous output state, so it inherits the dependency structure of the recurrence. A gate that is not exercised in the training distribution, because the sequences are too short or the boundaries are too clean, will be poorly calibrated on inputs where the situation does occur, and the state written at that point is wrong for the rest of the sequence.

Saturation is also inherited. If the pre-activation of a gate is driven far positive, the logistic function returns exactly one in the arithmetic being used, and its derivative is zero; the gate then cannot be moved by gradient at all, and the cell’s behaviour at that step becomes unconditional. Making gate inputs smaller, adding normalization inside the cell, or initializing the recurrent part of the gate matrices to be orthogonal all shift the starting point, but none of them prevents saturation from being reached on long sequences with large activations.

import numpy as np

def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))

def lstm_step(x, h, c, p):
    """One LSTM step. p holds Wf, Wi, Wg, Wo (each d x (n+d)) and bf, bi, bg, bo."""
    xh = np.concatenate([x, h])
    f = sigmoid(p['Wf'] @ xh + p['bf'])       # how much cell state survives
    i = sigmoid(p['Wi'] @ xh + p['bi'])       # how much candidate is written
    g = np.tanh(p['Wg'] @ xh + p['bg'])       # candidate content
    o = sigmoid(p['Wo'] @ xh + p['bo'])       # what is exposed as output state
    c = f * c + i * g
    return o * np.tanh(c), c

The code makes the parameter accounting visible: four matrices of shape $d \times (n+d)$, and a state that is two vectors rather than one. It also shows why the depth of the cell matters for inference latency, since each of the four products is on the critical path before the next step can begin, as does the sequence-level view in what a recurrent cell computes.

Recurrent dropout is orthogonal to gating and is applied to the gates rather than to the state, which keeps the cell’s memory path free of noise across steps; dropping state coordinates instead would destroy the information the gates were introduced to preserve. Layer normalization of the gate pre-activations is a separate change that addresses the saturation regime by controlling the scale of the inputs to the four maps.