Tag
batch-size
- Dec 1, 2024 Gradient Noise Scale and Critical Batch Size How the ratio of gradient variance to gradient squared norm sets the batch size beyond which additional examples buy no reduction in steps, and why that ratio rises over training. deep learning
- Dec 30, 2022 Gradient Accumulation: What Micro-Batching Preserves and What It Does Not Splitting a batch into micro-batches keeps the arithmetic of a large batch but changes normalisation, batch-dependent layers, and the timing of every state update. deep learning