Tag
compute-budget
- Aug 21, 2026 Compute-Optimal Allocation Between Attention and Feed-Forward Width How parameter and flop budgets split between attention and MLP blocks, why the split shifts with sequence length, and which allocations survive a change in context size. deep learning
- Feb 3, 2025 Why a Training Step Costs Three Times Its Forward Pass The parameter-times-token accounting behind training compute budgets, where the factor of three comes from, and when attention terms make the estimate wrong. deep learning