Attention as Content-Based Alignment Over Encoder States

A decoder that conditions on a single fixed-size vector has to compress everything the input contained into that vector before generation begins. Attention removes the compression step from that path: the encoder keeps a state for every input position, and each decoding step computes a weighted average over those stored states, with weights that depend on what the decoder is currently looking for. The mechanism is a lookup implemented with differentiable soft selection, and the consequences for gradient flow, cost, and what the model can express follow from that description.

From one vector to a weighted average

The decoder produces a query vector $q_t$ from its own state at step $t$, typically by a learned linear map. Each encoder position $i$ contributes a key vector $k_i$, also a linear map of the stored state. A score $e_{ti}$ is computed for every pair, the scores for a fixed $t$ are normalized with a softmax over $i$, and the context vector is the weighted sum $\sum_i \alpha_{ti} v_i$ of the encoder states, possibly after another linear projection to values $v_i$. The context is then concatenated to the decoder input or state and used to produce the output distribution.

Three details in that description decide the cost. The score matrix has $T_{\text{tgt}} \times T_{\text{src}}$ entries, so the arithmetic for scoring and normalizing grows as the product of the two lengths, $O(T_{\text{src}} \cdot T_{\text{tgt}} \cdot d)$ including the projections. The encoder states must all be retained, at $O(T_{\text{src}} \cdot d)$ memory, which is more than the $O(d)$ a recurrent encoder needs to keep its final state and less than the $O(T_{\text{src}} \cdot T_{\text{tgt}} \cdot d)$ that would be needed to store the full weight matrix if the decoder steps were computed in parallel. The weights themselves are $T_{\text{src}}$ numbers per decoding step, and they are a useful diagnostic: they say, for each output position, which input positions the model attributed the decision to.

Scoring functions

The additive form computes $e_{ti} = w^{\top} \tanh(W_q q_t + W_k k_i)$, which is a small feedforward network over each pair and works with query and key widths that differ. The multiplicative form computes $e_{ti} = q_t^{\top} k_i$, which requires equal widths and reduces the whole score matrix to a single matrix product; dividing by $\sqrt{d_k}$ keeps the variance of the dot product from growing with the width, since a sum of $d_k$ products of roughly unit-variance terms has variance proportional to $d_k$, and a softmax over scores with large variance saturates toward a one-hot distribution with vanishing gradients.

The trade between the two is arithmetic against flexibility. Multiplicative scoring makes the alignment a single dense matrix product that hardware executes efficiently and that can be batched over all decoder positions at once; additive scoring costs an extra matrix product per pair-evaluation and a nonlinearity, which is why it is used when the query and key spaces need separate projections or when the scores must be bounded. In both cases the number of parameters added is small compared with the recurrent layers, a few $d \times d$ matrices, and the scores are computed fresh at every decoding step because the query changes.

The cost also depends on how the decoder is run. Training evaluates all target positions at once, so the score matrix is materialized for the whole pair of sequences and holds $T_{\text{src}} \cdot T_{\text{tgt}}$ entries per attention operation. Inference with a single hypothesis needs one row of that matrix at a time, $T_{\text{src}}$ entries, but the stored encoder states remain $O(T_{\text{src}} \cdot d)$ for as long as generation continues. The asymmetry explains why the mechanism is cheap to train and comparatively expensive to serve on long inputs: the training pass is one batched matrix product, while serving pays for the retained states at every step.

Why the gradient path is short

The derivative of the context vector with respect to a stored state $v_i$ is the attention weight $\alpha_{ti}$ itself, an elementwise factor rather than a product of matrices. Nothing in that path depends on the distance between position $i$ and the current decoding step, so a signal from a decoder position to an encoder position a thousand steps away is scaled by one weight instead of by a product of a thousand Jacobians. That is a structural difference from the recurrence, whose long-range attenuation is described in backpropagation through time, and it holds even though the query itself was built by a recurrence.

The gradient with respect to the weights of the scoring function is a different matter. It flows through the softmax and through every score that contributed to the normalization, so an attention distribution concentrated on one position produces small learning signal for the others. The very property that makes alignment interpretable, that a peak weight means the model selected a position, is also the state in which the mechanism is hardest to move.

Order, permutation, and what the sum cannot see

The context vector is a weighted sum, and a sum over a set does not depend on the order of its terms. If the stored states carried no positional information, permuting the input positions and their states together would leave every context vector unchanged, and the model could not distinguish a sequence from any rearrangement of it. Recurrent encoders avoid the problem because each state was produced by a recurrence over a specific order and therefore depends on the positions before it; a stored state is a function of the prefix it was built from, so the set is ordered in effect. Any encoder whose states are computed positionwise independently must add positional information explicitly for the sum to be order-sensitive.

Alignment also has structure that the weights alone do not express. The softmax produces a distribution over all input positions at each step, so the mapping between output and input positions is a set of real-valued weights rather than an ordering. Nothing in the formulation prevents a decoder step from attending to positions in a scattered pattern that no monotonic alignment would produce, and nothing encourages it to move through the input in order. The distribution is learned from the objective, and for tasks where the alignment is genuinely many-to-one or one-to-many, such as a phrase in one language realised by several words in another, the flexibility is what allows a single mechanism to cover the cases.

Hard and constrained variants

Replacing the softmax-weighted average with a sampled discrete position makes the alignment a choice rather than a mixture, which sharpens the representation and matches tasks where a single input position is the right answer, but sampling is not differentiable with respect to the selection, so the model must be trained through a score-function estimator or a relaxation. The attention variant used for image captioning, where each step selects one region, is the canonical case.

Constraining the weights to move monotonically through the input is the other direction, and it is what makes streaming possible: if the alignment can only advance, the decoder can emit output while consuming input, and the memory of stored states stays bounded because old positions are never revisited. The same restriction underlies alignment mechanisms that are trained directly from an output sequence without a pre-specified correspondence, as described in CTC and monotonic alignment. The restriction costs expressiveness: a task whose alignment genuinely moves backwards cannot be represented at all.

Failure modes in the alignment distribution

Three failure patterns appear in the distribution rather than in the loss. The first is uniform attention, where the scores are nearly equal and the context becomes an unweighted average of the encoder states; this happens early in training when the scoring parameters are small and the softmax has little to amplify. The second is over-concentration, where one weight is close to one and the rest are below the precision of the arithmetic, so the context is effectively a single stored state and the mechanism behaves like a hard lookup with no gradient to the alternatives. The third is attention to positions that should not exist: padded input positions reach the scores unless they are excluded before the softmax, and the correction described in padding and masking is part of the mechanism, not an optimization.

Cost scales with the product of the two lengths, and it is paid at inference as well as training. For short inputs and wide states the alignment matrix is cheap relative to the recurrent arithmetic of $O(T \cdot d^{2})$; for long inputs against short outputs it becomes the dominant term. Deciding which regime a task is in precedes deciding whether to keep the recurrence at all.