Topic domain
Deep Learning
Neural architectures and the mechanics of training them.
- Articles
- 80
- Clusters
- 8
- Foundational
- 24
- Intermediate
- 32
- Advanced
- 24
Layers, activations, and backpropagation.
- The Fully Connected Layer as an Affine Map
- Pointwise Nonlinearity and the ReLU Family of Activations
- Reverse-Mode Differentiation on the Computational Graph
- Softmax Cross-Entropy and Its Collapsed Gradient
- Saturating Activations and Gradient Flow Through Depth
- Dropout Masks, Train-Time Scaling, and Inference Behaviour
Initialization, normalization, and schedules.
- Variance Preservation in Weight Initialization
- Batch Normalization and the Train-Eval Gap
- Learning Rate Warmup and the Early Instability Window
- Layer Normalization and the Residual Stream
- Decoupled Weight Decay and Adaptive Scaling
- Gradient Clipping by Norm and Loss-Scale Rescaling
Local connectivity and visual features.
- Cross-Correlation and Convolution in the Forward Pass
- Translational Weight Sharing and Feature Map Aliasing
- Receptive Field Arithmetic and Effective Coverage
- im2col Expansion, FFT Convolution, and Winograd Transforms
- Pooling as a Downsampling Operator and Its Signal Effects
- Transposed Convolution and the Checkerboard Artifact
Recurrence, attention, and long-range structure.
- Fixed-Window Models and the Hidden State They Cannot Represent
- What a Recurrent Cell Computes at Each Time Step
- Backpropagation Through Time and the Product of Jacobians
- Gates in LSTM and GRU Cells: What Each Gate Controls
- Bidirectional and Stacked Recurrence: Cost and Causality
- Padding, Packing, and Masking for Variable-Length Sequence Batches
Transformers
10Attention mechanics and scaling behavior.
- Why Scaled Dot-Product Attention Divides by the Square Root of the Key Dimension
- Multi-Head Attention as Independent Subspace Projections
- Permutation Equivariance in Self-Attention and What Position Encodings Restore
- Causal Masking, Additive Mask Terms, and Where the Mask Enters the Softmax
- Rotary Position Embeddings and Why the Score Depends Only on the Offset
- KV Cache Arithmetic: Bytes per Token, Layers, and the Batch Size Ceiling
Latent variable and diffusion approaches.
- The Evidence Lower Bound as a Latent Variable Objective
- The Reparameterization Trick and Gradient Flow Through Sampling
- Posterior Collapse and the KL Term That Wins
- Forward Diffusion as Closed-Form Gaussian Corruption
- Denoising Score Matching and the Noise-Prediction Objective
- Classifier-Free Guidance as an Interpolation of Two Predictions
Embeddings, contrastive methods, and transfer.
- Embedding Layers as Learned Lookup Tables
- Distance Concentration and Hubness in Embedding Spaces
- InfoNCE and the Temperature Parameter
- Data Augmentation as an Invariance Specification
- Momentum Encoders and Stop-Gradients
- False Negatives and Batch Composition
Mixed precision, sharding, and memory budgets.
- The Training Memory Budget: Parameters, Optimizer State, Gradients, Activations
- Choosing Between fp16 and bf16: Exponent Range and Mantissa Budget
- Why a Training Step Costs Three Times Its Forward Pass
- Gradient Accumulation: What Micro-Batching Preserves and What It Does Not
- Activation Checkpointing: Trading Recomputation for Stored Tensors
- Ring All-Reduce and the Cost of Averaging Gradients Across Devices
All Deep Learning articles
- Oct 16, 2026 Choosing Between fp16 and bf16: Exponent Range and Mantissa Budget Both formats are 16 bits, but they split those bits differently. The split decides whether a training run needs loss scaling and how much precision each weight update retains.
- Oct 12, 2026 Causal Masking, Additive Mask Terms, and Where the Mask Enters the Softmax Why a causal constraint is expressed as negative infinity added to the logits, how padding and causality combine in one layer, and which arithmetic the mask must precede.
- Oct 3, 2026 Layer Normalization and the Residual Stream How per-example statistics change the gradient path compared with batch statistics, why the placement of a normalization layer relative to a residual branch alters scale growth with depth, and what the learnable gains do.
- Sep 15, 2026 InfoNCE and the Temperature Parameter How the contrastive objective turns similarity scores into a softmax over the batch, why the temperature controls effective negative count and gradient shape, and what breaks when it is set wrong.
- Sep 6, 2026 Receptive Field Arithmetic and Effective Coverage How a unit back-propagates to the input positions that influence it, why the theoretical field overstates the real one, and how dilation and stride change the recurrence.
- Aug 26, 2026 im2col Expansion, FFT Convolution, and Winograd Transforms Three ways to turn a convolution into matrix arithmetic: what im2col replicates in memory, when the Fourier route wins, and what Winograd trades away in numeric range.
- Aug 21, 2026 Compute-Optimal Allocation Between Attention and Feed-Forward Width How parameter and flop budgets split between attention and MLP blocks, why the split shifts with sequence length, and which allocations survive a change in context size.
- Aug 17, 2026 Data Augmentation as an Invariance Specification Augmentations declare which transformations should not change a representation; the choice fixes what the embedding can ignore, and a mismatched set teaches the model to discard signal it needs later.
- Aug 17, 2026 Encoder-Decoder Sequence Models and the Teacher Forcing Assumption How a pair of recurrent networks maps a sequence to a differently sized sequence, what the fixed-size conditioning vector costs, and why training with gold prefixes diverges from inference.
- Jul 21, 2026 Linear Probing and Fine-Tuning as Different Measurements A frozen-feature linear probe and full fine-tuning answer different questions about a pretrained encoder, and the gap between their results is itself the informative quantity.
- Jul 16, 2026 Denoising Score Matching and the Noise-Prediction Objective How regressing on the noise that was added to a sample yields an estimate of the data score, and why the equivalence holds without any normalising constant.
- Jul 11, 2026 Cross-Correlation and Convolution in the Forward Pass What the sliding dot product computes, why deep learning libraries conjugate the kernel index, and how zero padding and output size arithmetic follow from the index bounds.
- Jul 6, 2026 Attention as Content-Based Alignment Over Encoder States How a decoder selects among stored states with a learned weight distribution, why that gives a short gradient path to any position, and what the attention weights cannot represent.
- Jul 3, 2026 Forward Diffusion as Closed-Form Gaussian Corruption How a fixed Markov chain turns any sample into noise in a fixed number of steps, why the cumulative transform is Gaussian in closed form, and what that buys for training.
- Jun 29, 2026 Pooling as a Downsampling Operator and Its Signal Effects What max and average pooling actually compute, why stride-based subsampling aliases, and how the backward pass routes gradient through a non-differentiable maximum.
- Jun 15, 2026 Jacobian-Vector and Vector-Jacobian Products in Autodiff Why differentiation machinery never materializes a Jacobian, the cost rule that decides whether to sweep forward or backward, and how the same primitives produce exact Hessian-vector products.
- May 25, 2026 Blockwise 8-Bit Optimizer State Without Losing the Second Moment Quantising Adam moments to eight bits cuts state memory by four, but small second-moment values quantise to zero. Blockwise scaling and dynamic exponent sharing fix that.
- May 21, 2026 Why Scaled Dot-Product Attention Divides by the Square Root of the Key Dimension Where the 1/sqrt(d_k) factor comes from, what quantity it holds constant as width changes, and how a saturated softmax destroys the gradient reaching queries and keys.
- May 19, 2026 The Evidence Lower Bound as a Latent Variable Objective How the intractable marginal likelihood of a latent variable model is split into a reconstruction term and a KL term, and what each term pushes the encoder to do.
- May 1, 2026 Ring All-Reduce and the Cost of Averaging Gradients Across Devices Why the ring algorithm moves a fixed multiple of the gradient size regardless of device count, and how bucketing hides the latency that the bandwidth term does not explain.
- Apr 28, 2026 Depth, Width, and the Linear Regions of ReLU Networks Every rectified network is a partition of its input space into cells where it computes an affine map, and the count of those cells grows polynomially in width and exponentially in depth.
- Mar 13, 2026 Alignment and Uniformity as Separate Objectives Contrastive losses can be decomposed into a term that pulls positive pairs together and a term that spreads all representations apart, and the two terms trade against each other in measurable ways.
- Feb 27, 2026 Decoupled Weight Decay and Adaptive Scaling Why adding a shrinkage term to the gradient is not the same as shrinking the parameter, how the adaptive denominator changes the effective decay per coordinate, and what the decoupled update fixes.
- Feb 25, 2026 Depthwise and Grouped Convolutions and the Cost per Channel How splitting channels into groups changes the parameter count, the arithmetic intensity, and the access pattern, and why a cheaper layer is often the slower one.
- Feb 15, 2026 Latent Diffusion and the Cost of Compressing Before Generating How moving the diffusion process into a compressed latent space changes the compute budget, what the autoencoder bottleneck removes, and which losses it silently introduces.
- Feb 5, 2026 Connectionist Temporal Classification and Monotonic Alignment How a blank symbol and a collapse rule turn frame-by-frame predictions into a sequence probability, why the sum over alignments needs a lattice recursion, and what the independence assumption costs.
- Jan 19, 2026 Embedding Layers as Learned Lookup Tables An embedding table is a matrix indexed by token id whose rows receive sparse gradients; its memory arithmetic, initialization, and weight tying decide what a model can learn from rare inputs.
- Jan 16, 2026 Residual Connections and the Identity Gradient Path How adding a block input to its output changes the Jacobian of a stack from a product of small factors into a sum that contains the identity, what that buys in depth, and where the argument stops holding.
- Nov 12, 2025 Momentum Encoders and Stop-Gradients Why contrastive and self-distillation methods train one branch with gradients and update the other by exponential moving average, and what the asymmetry buys in stability and target quality.
- Oct 8, 2025 Pipeline Parallelism: Filling the Bubble and Paying for It Micro-batch pipelining lets layers be split across devices, but the schedule leaves idle slots. How bubble fraction scales with depth and why interleaved schedules help.
- Sep 19, 2025 Multi-Head Attention as Independent Subspace Projections What the concatenation of heads buys over a single wide head, how the output rank is bounded, and why pruned heads can be removed without changing behaviour much.
- Sep 16, 2025 Reverse-Mode Differentiation on the Computational Graph How a recorded graph of elementary operations turns the chain rule into one reverse sweep, why values with several consumers force an accumulation, and where that sweep silently produces wrong gradients.
- Aug 25, 2025 Backpropagation Through Time and the Product of Jacobians How gradients travel backwards through an unrolled recurrence, why the product of per-step Jacobians decays or explodes, and what truncation and clipping change about the estimate.
- Aug 14, 2025 Variance Preservation in Weight Initialization How the Xavier and He scaling factors are derived from the fan-in of a layer, what breaks when the constant is wrong, and why depth turns a small gain error into a vanished signal.
- Jul 28, 2025 Distance Concentration and Hubness in Embedding Spaces Why nearest-neighbour search degrades in high dimensions, how a few points become hubs that absorb most queries, and what those effects imply for measuring similarity between learned vectors.
- Jul 7, 2025 Permutation Equivariance in Self-Attention and What Position Encodings Restore Why an attention layer treats a sequence as a set, the exact symmetry group it is invariant to, and how absolute, relative, and rotary encodings break that symmetry differently.
- Jun 11, 2025 Grouped-Query Attention and What Sharing Key Heads Costs How mapping several query heads onto one key head shrinks the cache by an exact factor, what the shared head cannot represent, and why the loss falls harder on keys than on values.
- Jun 9, 2025 Collapse and Redundancy Reduction in Self-Supervised Training Why self-supervised objectives tend toward constant or low-rank outputs, how whitening and decorrelation terms restore full rank, and what each approach costs in compute and stability.
- Apr 8, 2025 Gradient Checking by Finite Differences The truncation and round-off errors that bound how accurately a numerical derivative can be measured, how to choose the step size and the comparison metric, and why a check can pass on a broken derivative.
- Apr 7, 2025 Bidirectional and Stacked Recurrence: Cost and Causality How a reverse pass and additional layers widen what a recurrent network can see at one position, what each costs in latency and parameters, and when causality forbids the reverse pass.
- Mar 16, 2025 What a Recurrent Cell Computes at Each Time Step The exact arithmetic of an RNN cell, the tensor shapes and cost per step, why the recurrence is unrolled for training, and how hidden states behave when the sequence continues across batches.
- Feb 12, 2025 Feature Visualisation and What Convolutional Filters Reveal Optimising an input image against a single neuron, what regularisation each term contributes, and the evidence that early filters are oriented edge detectors.
- Feb 11, 2025 Gradient Clipping by Norm and Loss-Scale Rescaling What a per-tensor global norm clip bounds, how the scale factor is computed and applied, and why clipping interacts with the loss scale used in low-precision training.
- Feb 3, 2025 Why a Training Step Costs Three Times Its Forward Pass The parameter-times-token accounting behind training compute budgets, where the factor of three comes from, and when attention terms make the estimate wrong.
- Dec 26, 2024 Rotary Position Embeddings and Why the Score Depends Only on the Offset How rotating query and key pairs by an angle proportional to index makes attention a function of relative distance, and what the rotation periods cost when context grows.
- Dec 26, 2024 Stride-1 Backbones and Dense Prediction with Skip Connections Why segmentation decoders rebuild resolution that the encoder discarded, what a skip connection restores beyond detail, and how dilated backbones avoid the reconstruction entirely.
- Dec 1, 2024 Gradient Noise Scale and Critical Batch Size How the ratio of gradient variance to gradient squared norm sets the batch size beyond which additional examples buy no reduction in steps, and why that ratio rises over training.
- Nov 12, 2024 Cross-Modal Alignment and the Modality Gap Contrastive training can put matching images and text close together while the two modalities still occupy separate regions of the space; what produces that gap and what it costs for retrieval and composition.
- Oct 18, 2024 The Fully Connected Layer as an Affine Map What a dense layer computes, why the weight matrix is transposed and the bias broadcast, and how the parameter count, the backward pass cost, and the collapse of stacked layers follow from one identity.
- Oct 4, 2024 Cosine and Linear Decay Schedules as Step-Size Integrals What a learning rate schedule controls beyond the final value, how the shape distributes total parameter displacement across training, and why the choice between linear and cosine changes little at equal area.
- Sep 13, 2024 The Training Memory Budget: Parameters, Optimizer State, Gradients, Activations How device memory is consumed during training, why a model that fits for inference can be four times too large to train, and which term to cut first.
- Sep 6, 2024 Classifier-Free Guidance as an Interpolation of Two Predictions How a single conditional model can be pushed toward its conditional distribution by extrapolating away from its unconditional prediction, and what that costs.
- Sep 6, 2024 KV Cache Arithmetic: Bytes per Token, Layers, and the Batch Size Ceiling How the decode-time key and value cache is sized, why it scales with every layer rather than only the last, and how its growth sets the maximum batch on a fixed device.
- Sep 6, 2024 ZeRO Stages: Sharding Optimizer State, Gradients, and Parameters How the three partitioning stages divide training state across data-parallel ranks, what communication each one adds, and why stage 3 changes the residency model rather than just the arithmetic.
- Aug 23, 2024 False Negatives and Batch Composition Every off-diagonal entry in a contrastive batch is treated as a negative, including entries that are semantically identical; the resulting pressure shapes the space and is controlled by what goes into the batch.
- Aug 6, 2024 Noise Schedule Shape and the Signal-to-Noise Ratio Curve Why the choice of β schedule changes where a diffusion model spends its capacity, how variance-preserving and variance-exploding conventions differ, and when a cosine schedule beats a linear one.
- Jul 29, 2024 Composing a Parallelism Layout From a Memory and Bandwidth Budget Data, tensor, pipeline, and shard dimensions interact non-linearly. A procedure for choosing their sizes from per-device memory, interconnect tiers, and batch constraints.
- Jul 4, 2024 Loss-Scale Dynamics and Spike Detection in Mixed Precision Why a fixed loss scale cannot serve both the largest and smallest gradients in a run, how dynamic scaling detects overflow, and what happens to optimizer state when a step is skipped.
- May 17, 2024 Sliding-Window Attention and the Reach of a Stacked Local Window How a fixed-width window bounds per-layer cost while the stack multiplies its reach, what the reach formula predicts, and where the prediction fails because information cannot travel that far.
- May 9, 2024 Gradient Signal Propagation Through Depth at Initialization How the product of layer Jacobians determines whether a deep stack can be trained at all, why that product is easier to keep benign in residual architectures, and what the propagation analysis predicts about depth and width.
- May 9, 2024 Posterior Collapse and the KL Term That Wins Why a latent variable model can learn to ignore its latents entirely, how the KL term competes with reconstruction, and which interventions change the balance.
- May 8, 2024 Saturating Activations and Gradient Flow Through Depth How the derivative of a squashing function caps the signal that reaches early layers, why squaring the error makes the vanishing worst exactly when the network is most wrong, and which structural changes restore the signal.
- Apr 11, 2024 Bits per Dimension and Likelihood on Discretised Data Why a continuous density model of quantised images has a likelihood that depends on the quantisation step, how the change of variables works, and what the number does and does not compare.
- Mar 25, 2024 Activation Checkpointing: Trading Recomputation for Stored Tensors How discarding activations at segment boundaries and recomputing them in backward reduces memory, what the extra forward pass costs, and where the optimal segment count falls.
- Feb 17, 2024 Learning Rate Warmup and the Early Instability Window Why the first few hundred optimization steps are different from the rest, what large early updates do to adaptive optimizers, and how warmup bounds the damage.
- Jan 24, 2024 Beam Search Decoding: Scoring, Length Normalization, and Pruning How a decoder searches over sequences instead of choosing per step, why accumulated log-probabilities need a length term, and how beam width trades compute for coverage.
- Nov 28, 2023 Transposed Convolution and the Checkerboard Artifact How the backward pass of a convolution defines an upsampling operator, why unequal kernel overlap produces a fixed grid pattern, and how to avoid it entirely.
- Nov 6, 2023 The Reparameterization Trick and Gradient Flow Through Sampling Why a sampling operation blocks backpropagation, how the score-function estimator works around it, and what the reparameterized estimator changes about gradient variance.
- Nov 4, 2023 Translational Weight Sharing and Feature Map Aliasing How a kernel applied at every position implements a translation-equivariant prior, what the stride lattice does to that guarantee, and where shift invariance actually breaks.
- Oct 15, 2023 Batch Normalization and the Train-Eval Gap Why batch statistics at training time and running averages at inference time produce different outputs, and how that mismatch is exploited to estimate what a network actually learned.
- Sep 26, 2023 Padding, Packing, and Masking for Variable-Length Sequence Batches What it costs to batch sequences of different lengths behind a rectangular tensor, how masks keep padding out of losses and attention, and how packed sequences avoid the wasted arithmetic.
- Sep 7, 2023 Reverse-Time SDEs and the Probability-Flow ODE How the time-reversed diffusion process is defined by the score, why a deterministic ODE shares its marginals, and what the two formulations trade against each other.
- Aug 25, 2023 Pointwise Nonlinearity and the ReLU Family of Activations Why one elementwise nonlinear function is enough to break the collapse of stacked affine layers, how the rectifier behaves at its kink, and what makes a unit permanently dead.
- May 6, 2023 Activation Memory, Gradient Recomputation, and the Checkpoint Trade How the memory cost of a convolutional training step is computed, why activations outgrow weights, and what gradient checkpointing buys at what arithmetic price.
- Dec 30, 2022 Gradient Accumulation: What Micro-Batching Preserves and What It Does Not Splitting a batch into micro-batches keeps the arithmetic of a large batch but changes normalisation, batch-dependent layers, and the timing of every state update.
- Jun 13, 2022 Gates in LSTM and GRU Cells: What Each Gate Controls How multiplicative gates create an additive path through time, what the forget, input and output gates each multiply, and how a two-gate cell removes one of them.
- May 5, 2022 Fixed-Window Models and the Hidden State They Cannot Represent Why a predictor restricted to the last k inputs cannot count, track nesting, or carry information past its window, and what a learned hidden state changes about that limit.
- Apr 12, 2022 Softmax Cross-Entropy and Its Collapsed Gradient Why the gradient of the cross-entropy loss with respect to logits reduces to the prediction minus the target, what that saves computationally, and how to keep the exponentials from overflowing.
- Jan 13, 2022 Dropout Masks, Train-Time Scaling, and Inference Behaviour What the Bernoulli mask does to the expectation of an activation, why one convention scales during training instead of at test time, and how the mask changes what reaches the layers below.
- Nov 4, 2021 IO-Aware Attention: Tiling the Score Matrix Without Materializing It Why the intermediate attention tensors cost more than the arithmetic, how blockwise softmax stays exact, and what the online rescaling step has to preserve.