Range Calibration: Trading Clipping Error Against Rounding Error

The scale of an affine quantization is fixed by the range, and the obvious range is the observed one: take the minimum and maximum of the tensor and map them to the ends of the integer interval. That choice minimizes the maximum absolute error over the tensor, which sounds like the right objective until the distribution is examined. Activations in a trained network are not spread evenly to their extremes. They are concentrated near zero with a thin tail, and the tail is where the extremes come from. Setting the range to the extreme spends a large share of the code space on values that occur rarely, and degrades every value that occurs often.

Clipping the range below the observed maximum accepts a bounded error on the few clipped elements in exchange for a smaller step size on all the rest. The question is where to clip, and the answer depends on what the error is being measured against.

Two error terms that move in opposite directions

Fix a symmetric range $[-T, T]$ quantized to $b$ bits with $q_{\max} = 2^{b-1}-1$. The step is $s = T/q_{\max}$. Each value $x$ contributes

$$e(x) = \begin{cases} \text{rounding error, at most } s/2 & |x| \le T \ \text{clipping error, } |x| - T & |x| > T \end{cases}$$

Widening $T$ reduces clipping and increases $s$, so it increases rounding error everywhere. Narrowing $T$ does the reverse. There is an interior optimum for any distribution with a tail, and it sits strictly inside the observed range whenever the tail carries little probability mass.

For a concrete shape, take a Laplacian distribution with scale parameter $b_L$, which is a reasonable stand-in for post-activation values. Its density near zero is high and decays exponentially. If the range is set to the sample maximum over a few million elements, that maximum is roughly $b_L \ln(2N)$; for $N = 10^6$ that is about 14 times $b_L$. Clipping at $6 b_L$ instead discards values beyond six scale lengths, which carry a few parts in a thousand of the mass, and halves the step size. Rounding error, which applies to every element, halves; clipping error, which applies to a fraction of a percent, doubles. The mean squared error falls by roughly a factor of four, because the rounding term dominates by two orders of magnitude in the count of affected elements.

The reasoning is general: mean squared error is dominated by the term that applies to the most elements, not by the term with the largest single magnitude. A quantization scheme that minimizes worst-case error will always lose on average error, and inference accuracy tracks the average.

Percentile clipping and its blind spot

The simplest practical criterion is a percentile: clip at the value below which some fraction of observed elements fall, typically in the high nineties or above. This requires only a sorted sample or a histogram, both of which are cheap to compute during calibration, and it is insensitive to a handful of extreme values that would otherwise set the range.

The blind spot is that the percentile is chosen without reference to the integer grid. A percentile that lands between two grid points wastes part of the range, and a percentile chosen per tensor does not account for how the clipped elements are distributed: clipping the largest 0.1 percent of a tensor whose tail is a single channel’s outlier is a very different operation from clipping the largest 0.1 percent of a tensor whose tail is smoothly decaying. In the first case the clipped values are all in one place and the error is structured; in the second it is spread.

A second blind spot is that a percentile criterion is applied per tensor independently, so the resulting step sizes across layers are not comparable. Two layers with the same nominal bit width end up with different effective precision, and the layer with the wider distribution is the one that suffers. When the model is later given a per-layer bit assignment, the calibration criterion and the assignment criterion must agree, or the assignment optimizes the wrong quantity.

Divergence criteria and what they preserve

A different family of criteria chooses the threshold so that the distribution of quantized values resembles the distribution of original values. One formulation minimizes the Kullback-Leibler divergence between the two histograms, treating the quantization as a coarsening of the original distribution and asking which coarsening loses the least information.

The procedure is a search over candidate thresholds. For each candidate, the original histogram is truncated at the threshold and the overflow mass is folded into the last bin; the result is then expanded to the target number of bins and compared against the original with the divergence. The threshold with the smallest divergence wins. Because the search is over a one-dimensional family and the divergence is evaluated on a histogram, the cost is linear in the number of candidate thresholds and in the histogram size, and it runs once per tensor at calibration time.

What this criterion preserves is the shape of the distribution rather than the magnitude of the error. It tends to select thresholds that keep small values distinguishable, because the divergence is sensitive to the relative mass in each bin and the near-zero bins hold most of the mass. For activations that feed a subsequent layer’s dot product, that is usually the property that matters: the small values are numerous and each contributes a small product, while the large values are few.

The cost is that the criterion is defined on a histogram, so its result depends on the bin width, and it has no closed form. Two implementations that use the same criterion with different histogram resolutions can select different thresholds.

Choosing the criterion from the consumer

The error metric should match how the value is used, and the two consumers in a transformer block use activations differently.

An activation entering a matrix multiplication is multiplied by weights and summed. The contribution of a single activation error is its magnitude times the norm of the weight row it multiplies, and errors across the contraction dimension add in quadrature if they are independent. The relevant statistic is the mean squared error, and the relevant distribution is the one over the contraction axis. Clipping to minimize mean squared error over the tensor is the right criterion here, with the caveat that the error should be measured after the weight multiplication if the weights have very different norms per row.

For a symmetric range the minimum-error threshold can be found directly from a sorted sample. For each candidate threshold, the elements below it incur rounding error bounded by half the step and the elements above incur clipping error, and the two contributions are summed:

def mse_optimal_threshold(samples, qmax):
    """Symmetric int quantization to qmax levels; return the threshold
    minimizing the mean squared reconstruction error over the sample."""
    xs = sorted(abs(x) for x in samples)
    n = len(xs)
    best_t, best_err = xs[-1], float('inf')
    for i in range(1, n + 1):
        t = xs[i - 1]                      # candidate: clip everything above t
        step = t / qmax
        err = 0.0
        for x in xs[:i]:                   # rounding term, bounded by step/2
            q = round(x / step)
            err += (q * step - x) ** 2
        for x in xs[i:]:                   # clipping term, unbounded
            err += (t - x) ** 2
        err /= n
        if err < best_err:
            best_err, best_t = err, t
    return best_t

The quadratic scan is written for clarity; a prefix-sum formulation reduces it to linear time. The important property is that the minimum sits strictly below the largest sample whenever the tail is thin, and the amount by which it sits below is a function of how fast the distribution decays. For a distribution with a heavy tail the optimum moves toward the maximum, which is the regime where clipping is not worth performing.

An activation entering a nonlinearity is different. A value near a threshold of the nonlinearity, such as the boundary of a rectifier or the knee of a sigmoid, has an error that can flip its sign or its saturated state. Clipping error on a value that is already in the saturated region of a sigmoid changes nothing; rounding error on a value near the knee changes the output by a full step of the function. Calibrating the input of a nonlinearity to minimize mean squared error in the activation domain therefore does not necessarily minimize error in the output domain, and the threshold at which the nonlinearity bends is the place to look.

The practical consequence is that calibration statistics should be collected where the tensor is consumed, not where it is produced. A tensor that is stored, then normalized, then quantized has three candidate ranges and they are not the same range.

Statistics drift, and what to do about it

Ranges are computed from a calibration sample, and the sample is not the deployment distribution. The extremes of a distribution with a tail are exactly the statistic that is least stable across samples: the maximum of $N$ draws grows like $\sqrt{\log N}$ for a Gaussian tail, so a calibration set of a thousand examples and a production stream of a billion examples have different maxima by a noticeable margin.

There are three responses, and they are not exclusive. The first is to calibrate on a large and deliberately varied sample, accepting the cost of running it. The second is to compute the range from a statistic that is less extreme-sensitive, such as a high percentile combined with a safety factor, so that the range does not depend on the single largest observation. The third is to compute the range dynamically at inference time from the actual input, which removes the drift entirely and costs a reduction over the tensor plus the loss of a precomputable scale.

Dynamic per-tensor scaling is what most transformer inference stacks use for activations, and the reason is not only drift: the activation range in a transformer depends strongly on the input sequence, and a prompt with unusual token statistics produces activations outside any calibration range. A static range chosen from a calibration corpus is a bet that the deployment distribution resembles the corpus. The bet is usually fine for weights, which do not change, and often wrong for activations, which do.

Weights are the opposite case. They are fixed, so their range is fixed, and there is no drift. A clipped weight range is a permanent decision that can be evaluated offline against the layer’s output error, which makes weight calibration a static optimization problem rather than an estimation problem. Weight-only quantization exploits this: it can spend as much offline computation as it likes on choosing each weight’s representation, because the answer never has to be recomputed.

The interaction with what comes before and after

Clipping is not neutral with respect to the network around it. Clipping an activation introduces a bias: values above the threshold are all mapped to the same code, so the mean of the quantized tensor is lower than the mean of the original. That bias propagates into the next layer’s accumulator as a constant offset per channel, and it can be corrected by subtracting the expected clipped mass from the next layer’s bias. Whether the correction helps depends on whether the next layer is linear, which is why bias correction is applied after a linear layer and not after a normalization.

The correction is not always enough, because the error introduced by clipping is not only a mean shift; it also removes variance from the signal, and no additive term restores that. For layers whose output feeds a residual connection, the lost variance accumulates across the depth of the network, which is one reason residual-branch outputs are often kept at higher precision than the branch that feeds them.

A related asymmetry appears in the range chosen for a tensor that is the input to a layer whose weights have been scaled. Scaling a weight row by a constant and dividing the corresponding activation by the same constant leaves the product unchanged, but it changes both ranges. Equalization techniques move this scale between the weight and the activation to make both easier to quantize, and the calibration of each range must follow the move or it will be computed for the wrong tensor.

The last consideration is the interaction with the integer kernel. A range that produces a scale that is not a power of two requires a multiply to rescale the accumulator, and a range that produces a scale that is a power of two allows a shift. Rounding the chosen threshold to the nearest value that yields a power-of-two scale costs a small amount of range optimality and removes a multiply from the epilogue. Which side of that trade is better depends on whether the kernel is bound by arithmetic or by memory, and on hardware with fast integer multiply the difference may not be measurable at all.