Deep Learning · Cluster
Sequence Models
Recurrence, attention, and long-range structure.
- 01 Fixed-Window Models and the Hidden State They Cannot Represent Why a predictor restricted to the last k inputs cannot count, track nesting, or carry information past its window, and what a learned hidden state changes about that limit.
- 02 What a Recurrent Cell Computes at Each Time Step The exact arithmetic of an RNN cell, the tensor shapes and cost per step, why the recurrence is unrolled for training, and how hidden states behave when the sequence continues across batches.
- 03 Backpropagation Through Time and the Product of Jacobians How gradients travel backwards through an unrolled recurrence, why the product of per-step Jacobians decays or explodes, and what truncation and clipping change about the estimate.
- 04 Gates in LSTM and GRU Cells: What Each Gate Controls How multiplicative gates create an additive path through time, what the forget, input and output gates each multiply, and how a two-gate cell removes one of them.
- 05 Bidirectional and Stacked Recurrence: Cost and Causality How a reverse pass and additional layers widen what a recurrent network can see at one position, what each costs in latency and parameters, and when causality forbids the reverse pass.
- 06 Padding, Packing, and Masking for Variable-Length Sequence Batches What it costs to batch sequences of different lengths behind a rectangular tensor, how masks keep padding out of losses and attention, and how packed sequences avoid the wasted arithmetic.
- 07 Encoder-Decoder Sequence Models and the Teacher Forcing Assumption How a pair of recurrent networks maps a sequence to a differently sized sequence, what the fixed-size conditioning vector costs, and why training with gold prefixes diverges from inference.
- 08 Attention as Content-Based Alignment Over Encoder States How a decoder selects among stored states with a learned weight distribution, why that gives a short gradient path to any position, and what the attention weights cannot represent.
- 09 Connectionist Temporal Classification and Monotonic Alignment How a blank symbol and a collapse rule turn frame-by-frame predictions into a sequence probability, why the sum over alignments needs a lattice recursion, and what the independence assumption costs.
- 10 Beam Search Decoding: Scoring, Length Normalization, and Pruning How a decoder searches over sequences instead of choosing per step, why accumulated log-probabilities need a length term, and how beam width trades compute for coverage.