Activation Outlier Channels and Mixed-Precision Decomposition

Weights quantize gracefully. Their distributions are roughly bell-shaped, their extremes are modest relative to the bulk, and a per-channel scale handles what is left. Activations do not behave the same way in transformer language models, and the reason is structural rather than statistical. A small number of coordinates in the hidden state, often fewer than one in a hundred, carry values an order of magnitude or more above the rest. Quantizing those coordinates at the same scale as their neighbors either destroys the neighbors or clips the outliers, and no choice of range fixes both.

The response that works is not a better scalar quantizer. It is to stop treating the tensor as homogeneous.

Where a channel-scale outlier comes from

A transformer hidden state at a given layer is a vector of width $d$. After training, certain coordinates of that vector have a distribution with a heavy right tail, consistently across inputs. The phenomenon is not a property of one input; the same coordinates are extreme for nearly every token.

The mechanism is the residual stream. Every sublayer reads the residual stream, transforms it, and adds its output back. A coordinate of the residual stream accumulates contributions from many layers, and a coordinate that receives a consistent positive contribution from an attention head’s output projection grows without bound unless something counteracts it. Layer normalization counteracts the mean and the variance globally but is applied after the accumulation, so it rescales the whole vector rather than clipping individual coordinates. A coordinate that is systematically larger than its neighbors remains systematically larger after normalization.

The feed-forward block contributes too. Its first matrix projects the normalized hidden state into a wider intermediate space, and a row of that matrix with a large norm produces a large intermediate activation for every token. The second matrix projects back, and its columns determine which residual coordinates receive that contribution. A single large intermediate channel therefore deposits a consistent offset into a specific set of residual coordinates.

Once such coordinates exist, they are load-bearing. The model uses them to carry information across many layers, and they cannot simply be clamped: the magnitude is the signal. This distinguishes the outlier from a measurement artifact. A quantization scheme that suppresses it by rescaling the tensor globally also suppresses the information the model encodes in it.

Why the per-tensor scale cannot fix it

Take an activation vector in which one coordinate has magnitude 30 and the remaining 4095 coordinates have magnitudes below 1. A symmetric int8 scale chosen to cover 30 is about 0.236. A coordinate of 0.01 then maps to code 0, a coordinate of 0.5 maps to code 2, and the entire low-magnitude population, which is where most of the information sits, is represented by a handful of codes.

Per-token scaling does not help, because the outlier is present in every token’s vector, so the per-token maximum is always dominated by it. Per-channel scaling of activations would help, but activations are computed at run time and the per-channel scale would have to be a static property of the channel, which is a different mechanism: it works only if the outlier channels are stable across inputs, which they are, but it requires the kernel to apply a different scale per channel of the input vector, which breaks the single trailing rescale that makes the integer dot product cheap.

This is the crux. The outlier is real, it is stable, and it is incompatible with a uniform scale along the contraction axis. The three families of solutions all attack one of those three facts.

Decomposition: keep the outliers in high precision

The direct approach is to identify the outlier coordinates and compute their contribution separately, in floating point, while the remaining coordinates go through the integer path. The output is the sum of two matmuls: an integer matmul over the bulk of the contraction dimension and a floating-point matmul over the few outlier dimensions.

The decomposition is exact in the sense that the sum of the two products equals the product of the concatenated operands, so no approximation is introduced beyond the quantization of the bulk. The bulk quantization can then use a scale chosen for the bulk distribution, which is well behaved, and the outlier coordinates keep full precision.

def split_outlier_matmul(x, W, outlier_idx, quantize_int8):
    # x: [in_features], W: [out_features, in_features]
    keep = [i for i in range(x.shape[0]) if i not in outlier_idx]
    x_bulk = x[keep]
    W_bulk = W[:, keep]
    x_out = x[outlier_idx]
    W_out = W[:, outlier_idx]

    qx, sx = quantize_int8(x_bulk)          # per-tensor or per-token scale
    qW, sw = quantize_int8(W_bulk)          # per-output-channel scales
    y_bulk = (qx @ qW.T) * (sx * sw)        # integer matmul, float epilogue
    y_out = x_out @ W_out.T                 # float32, few columns
    return y_bulk + y_out

The cost is the second matmul and the gather. The gather is a strided copy of a few columns from a matrix, which on a row-major layout means touching many cache lines to read few elements. The float matmul has a contraction dimension of a few dozen at most, so its arithmetic is negligible, but its output is a full $[C_{\text{out}}]$ vector that must be added to the integer result, and it runs on the floating-point units that the integer path was supposed to avoid. For a layer with 4096 output features and, say, 32 outlier input features, the float matmul is under one percent of the multiply count but is a separate kernel launch and a separate accumulator.

Whether the outlier set is per-layer or global matters for the bookkeeping. If the set is determined once from calibration and stored in the checkpoint, the kernel can specialize on it, and the gather indices become part of the compiled graph. If the set is detected per input at run time, the kernel cannot be specialized and the gather becomes data-dependent, which costs more and complicates batching.

Migration: move the difficulty into the weights

The second family attacks the stability of the outlier rather than its magnitude. If a coordinate of the input vector is consistently large, the corresponding column of the weight matrix is consistently multiplied by a large value. Scaling that column down by a factor and scaling the activation coordinate up by the same factor leaves the product unchanged. Doing the reverse, scaling the activation coordinate down and the weight column up, reduces the activation range at the cost of widening the weight range.

The trade is asymmetric because the two tensors have different quantization difficulty. Activations are hard to quantize because of the outlier; weights are easy because per-output-channel scales already handle their spread. Moving magnitude from the activation side to the weight side therefore converts a hard problem into an easy one, up to the point where the weight column becomes the new outlier.

The scaling factor per channel is chosen so that both sides land within their respective ranges. A common rule sets the factor from the ratio of the activation channel’s maximum to the weight column’s maximum, raised to a power between zero and one that interpolates between “leave the activation alone” and “equalize the two maxima.” The power is a hyperparameter, and the fact that it must be tuned is a sign that the criterion is a heuristic rather than a derived optimum.

The attraction of migration is that it costs nothing at run time. The scales are folded into the weights and into the preceding operation that produces the activation, which for a transformer block is usually a normalization whose per-channel affine parameters absorb the factor exactly. The integer kernel is unchanged, there is no second matmul, and the accuracy improvement comes entirely from a better-conditioned pair of tensors.

The cost is that the preceding operation must be able to absorb the factor. A normalization with per-channel scale can; a residual add cannot, because it would change the value being added. In a transformer block, the activation entering a linear layer is usually the output of a normalization, which is why the technique fits so cleanly there, and the activation entering the attention score computation is not, which is where it fits less cleanly.

What neither approach fixes

Both approaches assume the outlier set is small and stable. When it is neither, the decomposition’s float matmul grows toward the size of the integer one, and the migration’s weight-side range grows until the weights need the same treatment the activations did. Neither degrades gracefully; they degrade into the unquantized computation.

The second assumption is that the outlier is a property of channels. In some models the extreme values are concentrated in specific tokens rather than specific channels, typically the first token of a sequence or a token whose embedding has an unusual norm. A per-channel treatment does nothing for a per-token outlier, and a per-token scale does nothing for a per-channel one. The two mechanisms are orthogonal and a scheme that handles both must combine them, which is why activation quantization recipes for transformers usually specify per-token dynamic scales for the range and a channel-level treatment for the persistent outliers.

The third is that the outlier’s contribution is large in the output. It is large in the input, but the weight column it multiplies may have a small norm, in which case the product is ordinary and the outlier’s apparent severity is an artifact of looking at the activation alone. Calibrating on the activation range rather than on the product distribution overstates the problem for exactly those channels where the weight column is small. Range calibration argues the general case for measuring error at the consumer; the outlier case is where that argument has the largest effect.

The design constraint that remains

Every scheme here is a response to one fact: a uniform grid along the contraction axis cannot accommodate a distribution with two separated scales. The decomposition separates the axes into two tensors with one scale each. The migration moves mass between tensors until one scale suffices for both. A third option, keeping the activations in higher precision and quantizing only the weights, sidesteps the problem by not quantizing the hard tensor at all, at the cost of the integer path, as the bandwidth argument for weight-only formats describes.

The choice between them is a choice about where to spend complexity: in the kernel, in the checkpoint, or in the precision. The kernel route costs latency per token. The checkpoint route costs a calibration procedure that must be rerun when the model changes. The precision route costs bandwidth. None is free, and the relative cost depends on the deployment target more than on the model.