Tag
backpropagation
- Sep 16, 2025 Reverse-Mode Differentiation on the Computational Graph How a recorded graph of elementary operations turns the chain rule into one reverse sweep, why values with several consumers force an accumulation, and where that sweep silently produces wrong gradients. deep learning
- Feb 3, 2025 Why a Training Step Costs Three Times Its Forward Pass The parameter-times-token accounting behind training compute budgets, where the factor of three comes from, and when attention terms make the estimate wrong. deep learning
- Mar 25, 2024 Activation Checkpointing: Trading Recomputation for Stored Tensors How discarding activations at segment boundaries and recomputing them in backward reduces memory, what the extra forward pass costs, and where the optimal segment count falls. deep learning
- May 6, 2023 Activation Memory, Gradient Recomputation, and the Checkpoint Trade How the memory cost of a convolutional training step is computed, why activations outgrow weights, and what gradient checkpointing buys at what arithmetic price. deep learning