Tag
activation-memory
- Sep 13, 2024 The Training Memory Budget: Parameters, Optimizer State, Gradients, Activations How device memory is consumed during training, why a model that fits for inference can be four times too large to train, and which term to cut first. deep learning
- May 6, 2023 Activation Memory, Gradient Recomputation, and the Checkpoint Trade How the memory cost of a convolutional training step is computed, why activations outgrow weights, and what gradient checkpointing buys at what arithmetic price. deep learning