Tag
memory-budget
- May 25, 2026 Blockwise 8-Bit Optimizer State Without Losing the Second Moment Quantising Adam moments to eight bits cuts state memory by four, but small second-moment values quantise to zero. Blockwise scaling and dynamic exponent sharing fix that. deep learning
- Sep 13, 2024 The Training Memory Budget: Parameters, Optimizer State, Gradients, Activations How device memory is consumed during training, why a model that fits for inference can be four times too large to train, and which term to cut first. deep learning
- Jul 29, 2024 Composing a Parallelism Layout From a Memory and Bandwidth Budget Data, tensor, pipeline, and shard dimensions interact non-linearly. A procedure for choosing their sizes from per-device memory, interconnect tiers, and batch constraints. deep learning
- Mar 25, 2024 Activation Checkpointing: Trading Recomputation for Stored Tensors How discarding activations at segment boundaries and recomputing them in backward reduces memory, what the extra forward pass costs, and where the optimal segment count falls. deep learning
- Feb 20, 2024 The KV Cache Memory Budget and What It Costs per Token Deriving bytes per cached token from attention shapes, then turning the freed capacity into a hard ceiling on concurrency and a floor on cost per token. inference systems