Encoder-Decoder Sequence Models and the Teacher Forcing Assumption

Mapping one sequence to another of a different length is not something a single recurrent layer does naturally. A layer that produces one output per input has an output length tied to the input length, and a layer that produces one output at the end has no way to emit a sequence at all. The encoder-decoder arrangement resolves this by splitting the roles: one recurrent network reads the input and produces a fixed-size conditioning vector, and a second recurrent network generates the output one symbol at a time, conditioned on that vector and on the symbols it has already produced. The split is what makes input and output lengths independent, and it is also the source of the arrangement’s main limitation.

The conditioning vector and its width

The encoder runs over the input and its final state, or its final cell state, is retained as a summary of width $d$. That vector, fixed in size regardless of whether the input was ten symbols or a thousand, is the only channel between the two halves. The decoder is initialized with it, or receives it concatenated to its input at every step, and the difference between those two arrangements is real: initializing the state lets the decoder transform the summary through its own dynamics, while concatenating at every step keeps the summary available later in the generation and makes it harder for later steps to drift away from it.

The cost of the arrangement is visible in the parameter count, which is roughly the sum of the two networks’ cells, and in the memory of the conditioning vector, which is $d$ values no matter how much input there was. For a source of length $T_{\text{src}}$ the encoder passes $T_{\text{src}} \cdot d$ state values through a recurrence of depth $T_{\text{src}}$, and only the last of them survives into decoding. Anything the final state did not retain is unrecoverable, and the gradient path from a decoder position back to an early encoder position traverses the full length of both sequences, which is the decay structure of backpropagation through time applied twice.

One cheap improvement to that path is to reverse the input sequence. With the source read in reverse, the first source symbols are adjacent to the last encoder steps, and the first decoder outputs depend on the earliest source symbols in many tasks, so the average distance between corresponding positions shrinks substantially and the information does not have to survive as many steps. It changes nothing about the model class, only the distance over which gradient and state information must travel.

Generating in a loop

The decoder produces symbols one at a time. At each step it emits a distribution over the output vocabulary, one symbol is selected from it, and that symbol’s embedding becomes the input at the next step. Generation stops at an end-of-sequence symbol or when a maximum length is reached. Because each step consumes the previous choice, the loop is serial: $T_{\text{tgt}}$ steps of latency, no matter how much parallel hardware is available, which is the same dependency that makes recurrent inference resistant to batching across time.

The loop also changes what the model computes. Training a decoder on the probability of one symbol given the input and the preceding true symbols is maximum-likelihood estimation of a set of conditional distributions, one per position, each of which is well defined. Selecting the highest-probability symbol at each step optimizes a different quantity: it maximizes each conditional probability independently, and the product of independently maximal conditionals need not be maximal among sequences. The distinction is what motivates search over sequences, covered in beam search decoding.

Two costs of the arrangement are structural rather than implementation-specific. Generation is serial in the output length, so the number of dependent decoder steps equals the number of emitted symbols no matter how much parallel hardware is available, and batching raises throughput without reducing that latency. The conditioning vector is a single value of width $d$ used by every decoding step, so a wider decoder does not widen the channel between the two sequences: the summary has to be wide enough to hold what the task requires, and no amount of decoder capacity compensates for a summary that is too small.

Depth allocation between the halves has an asymmetric effect. Encoder depth buys a better summary of the input and is paid only during training and prefill; decoder depth buys a more expressive transition between successive outputs and is paid once per generated symbol, so it is decoder depth that appears directly in serving latency. The usual split gives the encoder the larger share and keeps the decoder as shallow as the task tolerates, with the understanding that the reverse pass in the encoder is permitted precisely because the whole source is available before generation starts, as described in bidirectional and stacked recurrence.

Teacher forcing and the gap it leaves

Training uses the true preceding symbols as inputs, an arrangement called teacher forcing. It is the only way to train a sequence decoder in parallel over positions, since every step has a known input and the whole target sequence can be evaluated in one pass. The alternative, feeding the model its own previous predictions, makes step $t$ depend on the model’s behaviour at $t-1$, so the training loop becomes serial in the target length and the model is trained on its own output distribution.

The gap between the two regimes is the central issue. During training the decoder never sees an input that came from its own error distribution, and during inference every input is exactly that. A model trained with teacher forcing on inputs of length $T_{\text{tgt}}$ encounters a distribution over prefixes that shifts as generation proceeds, with the shift growing with the number of self-produced symbols. The compounding is arithmetic: if a decoder makes an error at a randomly chosen position with probability $\varepsilon$, the probability that a generated sequence of length $T$ contains at least one error behaves like $1 - (1-\varepsilon)^{T}$ in the independent approximation, so errors that are individually rare accumulate across a long generation.

Scheduled sampling addresses the mismatch by occasionally substituting the model’s own previous prediction for the gold symbol during training, with the substitution probability annealed over the course of training so that early steps see mostly gold inputs and later steps see more self-produced ones. The benefit is a decoder trained on inputs closer to what it will see at inference. The cost is that the training objective is no longer a clean conditional likelihood: the target loss is computed under an input distribution that depends on the current parameters, and the estimator is biased relative to the maximum-likelihood objective. The method trades a well-defined objective for a better-matched input distribution.

Teacher forcing also imposes a structural constraint on the loss. The decoder’s output at step $t$ predicts the target symbol at $t+1$, so the target sequence is shifted by one relative to the decoder inputs, and the loss is a mean over positions of the per-position cross-entropy. Positions are masked so that padding contributes nothing, with the same bookkeeping described in padding, packing, and masking, and the normalization has to divide by the number of real target positions rather than by the padded length.

Removing the middle

Everything difficult about the encoder-decoder arrangement follows from the single vector between the two halves. That vector is fixed in size while the amount of information in the input is not, so the encoder is forced to discard input detail, and which detail it discards is decided by the training objective rather than by the task. Two source positions that are far apart in the input but both relevant to one output position compete for space in the same $d$ coordinates.

The alternative is to keep every encoder state and let the decoder select among them at each generation step, which removes the fixed-size summary from the path between the two sequences and gives each decoder position its own view of the input. That mechanism, and the alignment distribution it induces over source positions, is the subject of attention as content-based alignment. The encoder-decoder loop and the teacher-forcing assumption remain in place underneath it; what changes is only what the decoder is allowed to look at while generating.