Affine Quantization: Scale, Zero Point, and the Error They Introduce

A float32 tensor can hold roughly four billion distinct values, spread across a dynamic range that spans from about $1.2 \times 10^{-38}$ to $3.4 \times 10^{38}$ in the normal range. An 8-bit integer tensor holds 256. Reducing one to the other is not compression in the entropy-coding sense; it is a change of number system, and everything that follows in this cluster is a consequence of the map between the two.

The map is affine, and it has exactly two free parameters. Getting those two parameters right for a given tensor is most of the practical work of quantization; the arithmetic that consumes them is comparatively mechanical.

The map, and why it needs a zero point

Write the real value as $x$ and the stored integer as $q$. The reconstruction is

$$x \approx s,(q - z)$$

where $s$ is a positive scale and $z$ is the zero point, an integer offset. Inverting the map for storage gives $q = \mathrm{clamp}(\mathrm{round}(x/s) + z,\ q_{\min},\ q_{\max})$. The grid of representable real values is therefore uniform, with spacing $s$, anchored so that the integer $z$ lands exactly on the real value 0.

That anchoring is the entire purpose of $z$. A symmetric format drops it by forcing $z = 0$ and restricting the integer range to $[-127, 127]$, which wastes one code on the negative side and cannot represent an asymmetric range without also wasting the codes at one end. An asymmetric format keeps $z$ and uses the full $[0, 255]$ or $[-128, 127]$ interval.

Zero matters because zero is not an ordinary value in a neural network. It is the padding value in a convolution, the value produced by a rectified activation for half its inputs, and the value that a pruned weight takes. If zero is not exactly representable, every padded position contributes a nonzero amount to a convolution sum, and every inactive activation leaks a constant into the accumulator. Those contributions are small individually and systematic in aggregate, which is worse than being large and random: a systematic offset can be folded into a bias term, but only if the implementation knows it is there.

The scale is fixed by the range: for an asymmetric format, $s = (x_{\max} - x_{\min})/(q_{\max} - q_{\min})$, and then $z = q_{\min} - \mathrm{round}(x_{\min}/s)$, clamped to the integer interval so that a range that does not align to the grid does not push $z$ out of bounds. The clamp is not cosmetic. Without it, a range whose minimum falls just below a grid point produces a $z$ one below $q_{\min}$, and the reconstruction of the minimum then requires an integer that cannot be stored.

def affine_params(lo, hi, qmin, qmax):
    if hi <= lo:                      # degenerate range: all values identical
        return None, None
    s = (hi - lo) / (qmax - qmin)
    z = qmin - round(lo / s)
    return s, min(max(z, qmin), qmax)

def quantize(x, s, z, qmin, qmax):
    q = round(x / s) + z
    return min(max(q, qmin), qmax)

def dequantize(q, s, z):
    return s * (q - z)

What the uniform grid does to the error

Rounding to the nearest grid point leaves an absolute error no larger than $s/2$. That bound is uniform across the tensor, which is the appealing property of the affine format and also the source of its central weakness.

The relative error is $s/(2|x|)$, which grows without bound as $x$ approaches zero. A tensor whose largest magnitude is 4.0 quantized to 8 bits over the range $[-4, 4]$ has $s \approx 0.031$, so a weight of $0.001$ carries an error of up to 0.0155 against a value of 0.001. The absolute error is tiny; the relative error is over 1500 percent. Floating-point formats do not have this property, because their spacing scales with the exponent. A float32 value near $10^{-3}$ is represented with the same 24-bit significand as a value near $10^{3}$.

Whether that matters depends on what the value feeds. A weight multiplied by an activation contributes its absolute error times the activation magnitude, so the relevant question is not the relative error of the weight but the absolute error against the scale of the products in the same sum. A small weight contributes a small product whether or not it is accurately represented. An activation that feeds a division, a normalization, or a logarithm is a different case, which is why the same bit width is routinely applied to weights and much less routinely to the inputs of a normalization layer.

The second property of the grid is that the error is deterministic and strongly correlated with the input. Rounding is a sawtooth function of $x/s$: the error rises linearly, drops by one step at each half-integer, and repeats. Successive elements of a tensor that are close in value receive nearly identical errors, so the error does not average out across a sum the way independent noise would. It averages out across a sum only when the summed terms are spread over many periods of the sawtooth, which is the case for a dot product over thousands of terms with varied magnitudes, and not the case for a short sum of similar values.

A rough accounting: 8 bits gives 256 levels across the range, so about 8 bits of magnitude resolution, against the 24-bit significand of the source format. The reduction is roughly 16 bits of mantissa. The compensation is that the values being represented occupy a much narrower dynamic range than float32’s full extent, and that the range was chosen from the data rather than fixed by the format.

The four terms inside an integer dot product

The reason the zero point survives as a design choice rather than being optimized away is that integer hardware cannot subtract a per-element offset for free. Consider a dot product of $K$ quantized weights and quantized activations:

$$\sum_{i} s_w (q_{w,i} - z_w), s_a (q_{a,i} - z_a)$$

Pulling the scales out, which is valid because they are per-tensor constants, leaves

$$\sum_i q_{w,i} q_{a,i} ;-; z_a \sum_i q_{w,i} ;-; z_w \sum_i q_{a,i} ;+; K z_w z_a$$

Only the first term is a genuine integer dot product. The second is $z_a$ times the row sum of the weights, a quantity that does not depend on the input and can be computed once when the weights are loaded. The third is $z_w$ times the sum of the activations over the contraction dimension, which must be computed per input but is a single reduction rather than a full product. The fourth is a scalar that depends only on the two zero points and the contraction length.

An implementation that precomputes the weight row sums and folds the constant term into the layer bias is left with one extra reduction over the activations per forward pass. That is the price of asymmetry, and it is small. It is also the reason symmetric weight quantization with $z_w = 0$ is preferred wherever the weight distribution permits it: it deletes two of the four terms outright.

Accumulator width follows from the same expression. With 8-bit unsigned activations bounded by 255 and 8-bit signed weights bounded by 127, a single product is at most 32385, and a sum of $K$ products fits in signed 32-bit arithmetic as long as $K < 2^{31}/32385$, which is about 66,000. Any realistic contraction length is far below that, so int32 accumulation is sufficient for 8-bit operands without overflow checks. Wider operands or longer sums break the argument, which is why the accumulator width is chosen together with the bit width rather than independently.

Failure modes that the two parameters create

The degenerate range is the first. If every value in a tensor is identical, $x_{\max} = x_{\min}$ and the scale formula divides by zero. Implementations either clamp the scale to a small positive constant or mark the tensor as constant and skip the multiply. The all-zero tensor is the common instance, and it is not rare: a pruned channel, a disabled expert, or a freshly initialized bias can all produce it.

Non-finite values are the second. A single NaN in the calibration statistics poisons both the minimum and the maximum, and because NaN compares false against everything, a naive min/max accumulation may propagate it or may silently ignore it depending on the order of comparison. Infinities clamp the range to the float limit, which then produces a scale so large that every finite value quantizes to the same code.

The third is the outlier, and it is the one that shapes the rest of the cluster. A single element far outside the bulk of the distribution sets $s$ for the entire tensor, and every other element loses resolution in proportion. The affine map has no mechanism to spend codes where the data actually is; it spends them uniformly over a range that one value chose. Range calibration is the set of techniques for choosing that range deliberately rather than accepting the extremes, and per-channel granularity is the structural fix: give each output channel its own pair of parameters so that one channel’s outlier cannot set another channel’s grid.

The fourth is rounding mode. round in the formulas above is ambiguous between half-to-even, half-away-from-zero, and truncation-with-bias-correction. The difference is one unit in the last place per element, which is within the error bound but not within bit-exact reproducibility. Hardware conversion instructions and library kernels do not all agree, so two implementations that compute the same mathematical function can produce different integer tensors from the same float input.

The last is the interaction between the scale and the consumer. A scale computed for a tensor that is later fused with a batch normalization, or concatenated with another tensor along the channel axis, is no longer the right scale for the fused result. Fusion changes the range, and a range computed before fusion is a range computed for a different tensor. That is a bookkeeping failure rather than a numerical one, and it is the most common way a quantized model that validates correctly in isolation degrades after graph optimization.

What the affine map buys, for two scalars per tensor, is a representation whose arithmetic is exact integer arithmetic with a known error bound. What it costs is a uniform grid applied to data that is rarely uniform. The remaining articles in this cluster are, in one way or another, about closing that gap: by moving the grid, by shrinking the region each grid covers, or by choosing the rounding direction rather than accepting the nearest one.