Choosing Between fp16 and bf16: Exponent Range and Mantissa Budget

A 16-bit float is 16 bits however it is divided. fp16 and bf16 both occupy two bytes per element, both move through the same memory buses at the same rate, and both are handled by the same tensor-core instructions on hardware that supports them. They differ in a single decision: how many of those 16 bits describe the exponent and how many describe the fraction. That decision propagates into dynamic range, precision per update, and whether the training loop needs a scaling mechanism at all.

The choice is not cosmetic. A format with an exponent range too narrow forces a compensating trick, and the trick has its own failure mode. A format with a mantissa too short silently discards small updates. Knowing the bit layout makes both consequences predictable.

The bit layout

IEEE 754 binary16, usually called fp16 or half precision, allocates 1 sign bit, 5 exponent bits, and 10 fraction bits. The 5-bit exponent is biased by 15, giving exponent values from $-14$ to $15$ for normal numbers. The largest finite value is $65504$; the smallest positive normal value is $2^{-14} \approx 6.1 \times 10^{-5}$, with subnormals extending down to $2^{-24} \approx 6.0 \times 10^{-8}$.

bfloat16 is not an IEEE 754 format. It allocates 1 sign bit, 8 exponent bits, and 7 fraction bits. The exponent field has the same width and bias as binary32, so bf16 covers exactly the range of fp32: normals from roughly $1.2 \times 10^{-38}$ to $3.4 \times 10^{38}$, with the same subnormal structure. The trade is explicit: three more exponent bits, three fewer fraction bits.

The mantissa width determines relative precision. With an implicit leading one, 10 stored fraction bits give 11 significant bits, so consecutive representable values near a power of two are separated by $2^{-10}$ of that power, about one part in a thousand. Seven stored bits give 8 significant bits, about one part in 256. In decimal terms fp16 resolves roughly three significant digits, bf16 roughly two and a half.

That difference matters most where small numbers are added to large ones, which is exactly what a weight update does.

Why the narrow exponent forces loss scaling

The fp16 dynamic range spans about $2^{30}$ in exponent, from $6 \times 10^{-8}$ to $65504$. Gradient magnitudes in a deep network routinely fall below the lower end. A gradient of $10^{-8}$ in fp16 is subnormal and loses precision; a gradient of $10^{-9}$ is zero. Once a gradient underflows to zero, the corresponding parameter receives no update for that step, and nothing in the loss signals that it happened.

Loss scaling compensates by multiplying the loss by a constant before the backward pass. By linearity, every gradient is multiplied by the same constant, moving the distribution up into the representable range. The optimiser then divides the gradients by the constant before applying them, or equivalently divides the learning rate. The scale must be chosen so that the smallest useful gradients stay normal while the largest do not overflow to infinity; because those two bounds are set by the data rather than the architecture, the scale has to be found at runtime.

The standard mechanism starts from a large power of two and adjusts. If any gradient in a step is non-finite, the step is skipped, the scale is halved, and the parameters are left untouched. If a chosen number of consecutive steps complete without an overflow, the scale is doubled. The power-of-two restriction keeps the scaling and unscaling exact: multiplying by $2^k$ changes only the exponent field, so no rounding is introduced.

bf16 needs none of this. Its exponent range is fp32’s, so a gradient that is representable in fp32 is representable in bf16, and the smallest normal bf16 is around $10^{-38}$, far below anything a gradient produces before the network has diverged for other reasons. Overflow is likewise impossible for values that fp32 could hold. The scale is effectively 1 and the mechanism disappears.

This is the main practical argument for bf16. It removes a stateful control loop from the training step, and with it the failure mode where a single anomalous batch drives the scale down and the run spends many steps climbing back.

Where bf16’s short mantissa hurts

The absence of scaling is paid for in precision. Accumulating a long sum in bf16 loses low-order bits quickly, which is why matrix multiply hardware does not accumulate in the input format. Tensor cores that accept bf16 or fp16 inputs accumulate products in fp32, and the accumulate step is where the precision of the result actually comes from. The input rounding is a single rounding per element; the accumulation is not.

The place where the short mantissa does bite is parameter update. For a value in $[1, 2)$ the gap to the next representable number is $2^{-7} \approx 0.0078$ in bf16 and $2^{-10} \approx 0.00098$ in fp16. Suppose a weight has magnitude $1.0$ and the optimiser applies a step of $0.001$. In bf16 that step is well under half a unit in the last place, so adding it to the stored weight rounds back to the original value: the gradient was computed, the optimiser ran, and the parameter did not move. In fp16 the step is just over half a unit and survives as a single increment, which is enough to keep the parameter moving but quantises every update to that grid. Neither format can be trusted to hold a parameter that accumulates many small updates, which is the reason both mixed-precision schemes keep a full-precision master copy of the weights. The optimiser state and the update arithmetic live in fp32; only the copy used for the forward and backward matmuls is narrowed. The memory cost of that master copy is accounted for in the training memory budget.

There is a second, subtler effect in bf16: because the relative precision is lower, the gradient itself is quantised more coarsely. For most training this is tolerable, since the gradient is an average over a batch and the noise in the estimate exceeds the rounding. It becomes visible when the batch is very small or when the loss is dominated by a few examples.

The cast boundaries in a step

A mixed-precision step is defined by where the dtype changes. A common arrangement:

# Parameters are stored in fp32; the compute copy is narrowed per step.
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
    logits = model(inputs)          # matmuls run in bf16, accumulate in fp32
    loss = criterion(logits, targets)

scaler.scale(loss).backward()       # a no-op scale for bf16, real for fp16
scaler.unscale_(optimizer)          # divide grads back before clipping
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
scaler.step(optimizer)              # skips the step if any grad is non-finite
scaler.update()                     # adjust the scale for the next step
optimizer.zero_grad(set_to_none=True)

Three details in that sequence carry weight. The cast happens at the boundary of each operation rather than to the tensors themselves, so the fp32 parameters are untouched and the narrowed copy is transient. Gradient clipping must run after unscaling, because the norm of a scaled gradient is the scaled norm and clipping against the unscaled threshold would clip everything. And the scaler must be consulted before the optimiser step, so that a non-finite gradient does not corrupt the parameters and the moments.

Operations that are numerically sensitive stay in fp32 by policy: softmax, layer normalisation, the loss computation, and the optimiser update. The autocast policy is a list of which operations are exempt, and the list matters because a softmax computed in fp16 can overflow when the exponent range is narrow, while the same softmax in bf16 is safe but less accurate than fp32.

Selecting a format

The decision reduces to two questions: whether the hardware supports the format at full throughput, and whether the training loop tolerates the narrower mantissa.

bf16 is the default choice on hardware that supports it natively, because it removes loss scaling and matches fp32’s range. fp16 remains relevant on hardware where it is the only accelerated 16-bit path, and it delivers slightly better precision per element when the scaling machinery is in place and working. Running both in the same model is possible but adds cast points without a clear benefit.

Formats below 16 bits change the analysis rather than extend it. An 8-bit float has too few mantissa bits for gradient computation but can be adequate for stored optimiser state, where the values are long-running averages and the precision requirement is lower; that trade is the subject of blockwise 8-bit optimiser state. The general rule holds: match the format’s exponent range to the quantity’s dynamic range, and its mantissa to the precision the update actually needs.