Requantization Inside the Kernel: From Int32 Accumulator to Int8 Output

A quantized convolution ends with an accumulator holding an int32 value whose units are not the units of the output tensor. The accumulator sums products of two integers, and its scale is the product of the input scale and the weight scale. The output tensor has its own scale, chosen for the next layer’s range. Converting between them requires a division by a real number, and doing that division in floating point inside a kernel defeats the purpose of quantizing at all. The conversion is therefore done with integer arithmetic, and the way it is done is the same in every mature quantized inference library.

The ratio that has to be applied

For a layer with input scale $s_x$, weight scale $s_w$, and output scale $s_y$, an accumulator value $a$ corresponds to the real number $s_x s_w a$ before the bias, and the output code should be

$$q = \mathrm{clamp}\left(\mathrm{round}\left(\frac{s_x s_w}{s_y}, a\right) + z_y,\ 0,\ 255\right)$$

The quantity $M = s_x s_w / s_y$ is the requantization multiplier. It is a positive real number, typically less than one because the output scale is usually larger than the product of the input and weight scales, but it is not bounded: a layer whose output range is narrow can have $M$ greater than one. It is also not a power of two, and forcing it to be one costs accuracy in a way that is measurable across a deep network.

So the kernel must multiply an int32 accumulator by an arbitrary positive real and round the result. Integer-only implementations do this by representing $M$ as a fixed-point pair: an integer multiplier $m$ and a right shift $n$, such that $M \approx m / 2^{n}$.

Choosing the fixed-point pair

The standard construction picks an integer $m$ in the 31-bit range and a shift $n$ such that $M \approx m/2^{n}$. The choice of $n$ is forced by the width: $n$ is the unique integer for which $M \cdot 2^{n}$ lands in $[2^{30}, 2^{31})$, which is to say the leading bit of the scaled multiplier sits just below the sign position. Then $m = \mathrm{round}(M \cdot 2^{n})$, and the approximation error is entirely the rounding of that final step. The shift $n$ is typically in the low thirties for multipliers near one, and grows as $M$ shrinks.

The 31-bit width is not arbitrary. A signed 32-bit multiply of two 32-bit values overflows into 64 bits, and the high part is what matters. Restricting $m$ to 31 bits means $m < 2^{31}$, and the product of $m$ with an accumulator bounded by $2^{31}$ fits in 62 bits, so a signed 64-bit intermediate holds it without overflow. The subsequent shift discards the low $n$ bits and the result is narrowed to 32 bits and then saturated to the output range.

The error introduced by this approximation is bounded by the quantization of $m$: an absolute error of at most half a unit in $m$, which relative to $m \ge 2^{30}$ is at most $2^{-31}$, about one part in two billion. For 8-bit outputs with 255 levels, that is far below one code, so the fixed-point multiplier is not a source of accuracy loss in practice. For 4-bit outputs with 15 levels the margin is still comfortable.

#include <stdint.h>

// Multiply a signed accumulator by a fixed-point multiplier m/2^n,
// rounding to nearest and saturating to the unsigned 8-bit output range.
static inline uint8_t requantize(int32_t acc, int32_t m, int n, int32_t zero_point) {
    int64_t prod = (int64_t)acc * (int64_t)m;
    // Round to nearest: add half the divisor before shifting (arithmetic shift
    // on a negative value then rounds toward negative infinity, which is
    // consistent as long as the added term has the same sign as prod).
    int64_t half = (int64_t)1 << (n - 1);
    prod += (prod >= 0) ? half : -half;
    int64_t shifted = prod >> n;              // arithmetic shift, floor
    int64_t out = shifted + zero_point;
    if (out < 0) return 0;
    if (out > 255) return 255;
    return (uint8_t)out;
}

The rounding detail in that function is the part most often wrong. A right shift of a negative two’s-complement value floors toward negative infinity, while adding a positive half rounds toward positive infinity for positive values and away from zero for negative ones only if the half term is subtracted. Mixing the two produces a result that is off by one code for negative accumulators, and the resulting error is a small negative bias in the output that accumulates across layers. Because the output is then fed into the next layer’s accumulator, the bias does not stay small.

Where the zero point enters and where it does not

The bias add happens in the accumulator domain, before requantization. A bias quantized with scale $s_x s_w$ and zero point zero is added directly to the int32 accumulator, which is why the bias must be stored at int32 precision even when the weights are 8-bit. A bias stored at 8 bits cannot represent the range of a sum of thousands of products.

The input zero point’s contribution is folded into the accumulator before the epilogue, as a per-output-channel correction. The output zero point is added after requantization, in the output integer domain. These two are different quantities with different scales, and a kernel that applies the output zero point to the accumulator or the input correction to the requantized value produces a systematically shifted output.

The per-channel case complicates the epilogue but not the inner loop. With a weight scale per output channel, the multiplier $M_j = s_x s_{w,j}/s_y$ varies per channel, so the kernel must carry a vector of $m_j, n_j$ pairs and apply the pair corresponding to the row it is finishing. If the shifts differ per channel, the loop cannot use a single vectorized shift; implementations therefore restrict $n$ to a per-tensor value and vary only $m$, which is possible when the scales are close enough that a common shift keeps every $m$ within the 31-bit bound. Choosing the common $n$ as the maximum required shift and left-shifting the smaller multipliers into place is the usual fix, at the cost of some precision in the multipliers that needed fewer bits.

Two-pass accumulation and the per-channel correction

The integer dot product over the contraction axis produces the accumulator; the correction for the input zero point requires the sum of the activations over that same axis, which is a second reduction. Doing both in one pass over the input is possible: the kernel accumulates $\sum q_x q_w$ and $\sum q_x$ simultaneously, and the second sum is shared across all output channels for the same input tile. That sharing is what makes the correction cheap. Computing it per output channel would repeat the same reduction $C_{\text{out}}$ times.

The weight row sum $\sum q_w$ has no such sharing: it differs per output channel and is a property of the weights, so it is precomputed at load time and stored alongside the bias. A kernel that recomputes it per inference is wasting a reduction that depends only on data that never changes.

Saturation, wraparound, and the accumulator width

The accumulator bound is the first correctness constraint. For 8-bit unsigned activations and 8-bit signed weights, the maximum product magnitude is $255 \times 128 = 32640$, and a sum of $K$ such products needs $\lceil \log_2(32640 K) \rceil$ bits. For $K = 4096$ that is about 27 bits, which fits comfortably in int32. For $K = 65536$ it is 31 bits, which does not, because the sign bit is consumed. A layer with a contraction dimension beyond roughly 65,000 cannot use int32 accumulation at 8 bits without risk, and implementations either widen to int64 for such layers or split the reduction into chunks that are summed in a wider type.

Wraparound is worse than saturation because it is silent. A kernel that overflows int32 produces a value with the wrong sign, and requantization then maps it to an extreme code that looks like a legitimate activation. Saturation at the output is visible in a range check; saturation of the accumulator is not. This is why some libraries clamp the accumulator periodically in long reductions, trading a compare per tile for the guarantee that the bound holds.

The second constraint is the negative range. An 8-bit signed activation can reach $-128$, which makes the product $128 \times 128 = 16384$ for a weight of $-128$; the bound is symmetric enough that the analysis above holds with 8 bits of headroom. An asymmetric activation range that reaches 255 on the positive side and 0 on the negative has its zero point somewhere in the middle, and the products are offset by the zero point, which is why the correction term is needed and why its magnitude must be included in the bound.

Bit-exactness across implementations

The specification for a quantized model usually defines the arithmetic in terms of real numbers, with the integer implementation as an approximation. Two implementations that follow the same specification can differ by one code in the output for a given input, because the rounding of the multiplier, the rounding of the requantization shift, and the order of the bias add are all unspecified in the real-number formulation.

For a single layer that difference is negligible. Across a deep network it can be amplified: a one-code difference at the input of a layer changes its accumulator by up to the sum of the weight magnitudes in the affected column, which can exceed one code at the output. Models that are sensitive to this, typically small models with tight tolerances or models evaluated with exact-match metrics, are affected by implementation changes that are numerically irrelevant in the large.

Libraries that care about this define the integer path precisely, including the multiplier construction, the rounding rule, and the saturation points, so that any conforming implementation produces the same bytes. The cost of that precision is that the specification constrains optimizations: a kernel cannot use a fused multiply-add that changes the intermediate rounding, or a different accumulation order that changes the overflow behaviour, without leaving the conforming set.

That constraint is the reason quantized kernels are usually written against a narrow set of arithmetic primitives rather than being free to use whatever the hardware offers. The requantization step described here is small, a handful of instructions per output element, but it sits at the boundary between the integer world and the next layer’s integer world, and every choice in it propagates. The rounding rule chosen here determines whether the clipping threshold chosen during calibration actually produces the error it predicted, because a biased rounding rule adds a systematic term the calibration did not model.

For very low bit widths the epilogue becomes a larger share of the work: at 2 bits, the inner loop does a quarter of the multiplies per output, while the requantization and packing cost per output is unchanged. That asymmetry is one of the reasons sub-4-bit formats are usually paired with a different kernel structure rather than a scaled-down version of the 8-bit one, and it connects to the rounding-direction techniques that change what the epilogue has to do.