Per-Channel and Group Scales: Granularity Against Metadata Cost

A weight matrix for a linear layer is not one distribution. It is a collection of rows, one per output feature, and each row has its own mean, its own spread, and its own susceptibility to outliers. A single scale for the whole matrix forces every row onto a grid sized by the widest row in the matrix, and rows with a fraction of that spread lose most of their codes to representing magnitudes they never produce. The fix is obvious once stated: compute the scale where the variation is. The engineering is in deciding where that is, and in paying for it.

Where the variation actually lives

Consider the matrix multiplication $y = Wx$ with $W$ of shape $[C_{\text{out}}, C_{\text{in}}]$. Every output element $y_j$ is a dot product of row $j$ of $W$ with the input vector. If row 3 has a maximum magnitude of 0.02 and row 900 has a maximum magnitude of 4.0, a per-tensor scale of $4/127 \approx 0.0315$ assigns row 3 about 0.6 of a code across its entire range. That row becomes effectively zero, or a coarse two-level signal, and the output feature it produces is destroyed while the other rows are fine.

Per-channel quantization gives each row $j$ its own scale $s_j$ and, for asymmetric formats, its own zero point $z_j$. Row 3 then gets a scale around $0.02/127$, and its 8 bits cover its own range with the same resolution the wide row gets over its range. The relative error per row becomes uniform instead of the absolute error being uniform.

The same reasoning applies along the input dimension in the other direction, and it is here that the choice stops being arbitrary. The contraction axis is the axis along which products are summed. Quantizing $W$ with a scale per row means the accumulator for output $j$ sums terms that all share $s_j$, so the scale factors out of the sum exactly once. Quantizing with a scale per column would mean each term in the sum carries a different scale, and the sum would require either a rescale per term, which defeats the integer kernel, or an approximation that adds a scale ratio to every product. Per-row is the only granularity that leaves a plain integer dot product with a single trailing rescale.

This is why per-channel weight quantization along the output axis is nearly free: it changes nothing about the inner loop, only the number of scales the loop consults at the end. It is also why the corresponding choice for activations is more delicate. Activations arrive as a vector with one scale per tensor or per token, and there is no output-axis structure to exploit; the analogue is per-token scaling, discussed with the KV cache in per-token scale calibration.

The metadata arithmetic

The cost of granularity is easiest to see as a ratio. Take a weight matrix with $N$ parameters stored at $b$ bits each. The payload is $Nb/8$ bytes. Per-tensor parameters add one scale, 4 bytes. Per-channel parameters along the output axis add $4 C_{\text{out}}$ bytes for a symmetric scale, or $8 C_{\text{out}}$ when a zero point is stored too. Group-wise parameters with group size $G$ along the input axis add $4 \cdot C_{\text{out}} \cdot C_{\text{in}}/G$ bytes at fp32.

For a square matrix of 4096 by 4096 at 4 bits: the payload is $16.7$ million parameters times 0.5 bytes, about 8.4 MB. Per-channel metadata is 4096 scales of 4 bytes, 16 KiB, which is under 0.2 percent of the payload. Group-wise with $G = 128$ multiplies the metadata by 32, to 512 KiB, about 6 percent of the payload. Group-wise with $G = 32$ is 24 percent. In bit-rate terms the payload rises from 4.0 to 4.25 at $G=128$ with fp32 scales, or to 4.125 with fp16 scales; at $G=32$ the same two choices give 5.0 and 4.5.

def effective_bits(payload_bits, rows, cols, group, scale_bytes, per_channel=False):
    """Total storage cost per weight, in bits, for a given scale granularity."""
    payload = rows * cols * payload_bits
    count = rows if per_channel else rows * cols // group
    return (payload + count * scale_bytes * 8) / (rows * cols)

# 4-bit weights, 4096x4096, fp16 scales
for g in (128, 64, 32, 16):
    print(g, round(effective_bits(4, 4096, 4096, g, 2), 3))
# per-channel instead of per-group
print('per-channel', round(effective_bits(4, 4096, 4096, 1, 2, per_channel=True), 3))

That ratio is the entire trade. Smaller groups give finer resolution and cost metadata linearly in $1/G$. The break-even point depends on how much error the finer scale actually removes, and that depends on the distribution inside the group.

There is a second, less obvious cost. The kernel must load the scale for a group before it can use the group. A dot product over $C_{\text{in}} = 4096$ with $G=128$ consults 32 scales; with $G=32$ it consults 128. If the scales are interleaved with the weights in memory, each group boundary is a separate small load, and the loads are not necessarily coalesced with the weight fetches. The arithmetic intensity falls because the same weight bytes now require more accompanying scale bytes to be fetched, and on a memory-bound decode step that is the number that matters, not the theoretical bit width.

A layout that packs the scales for a whole tile contiguously, so that one cache line covers the scales for many groups, keeps this from mattering. A layout that stores each scale adjacent to its group’s weights inserts a 2-byte gap every 64 or 128 bytes and defeats vectorized loads. The same quantization scheme therefore has different performance depending on a decision made by whoever wrote the packing routine.

Two-bit groups and the format zoo

Below 8 bits the scale overhead becomes the dominant term, and the formats that exist at 4 bits and below are largely distinguished by how they handle it. A 4-bit weight with a group size of 32 and an fp16 scale spends 0.5 bytes per weight plus 0.0625 bytes of scale, an effective 4.5 bits. Drop the group to 16 and the effective rate is 5 bits. At 2 bits with $G=16$, the scale is a quarter of the payload.

Two responses exist. The first is to quantize the scales themselves, storing them at 8 bits relative to a per-channel super-scale, which recovers most of the precision at a quarter of the cost. The second is to abandon per-group scales for a shared codebook, where a group of values is approximated by a small set of centroids and the stored data is an index into that set. A codebook of 16 entries for a group of 16 values is 4 bits per value with a table of 16 floats amortized over the whole tensor, and the table can be refined per group at a modest extra cost. This is the vector-quantization branch of the family, and the trade it makes is different: the grid is no longer uniform, so the reconstruction requires a lookup rather than a multiply, and the error is no longer bounded by half a step.

The metadata can also be eliminated by construction. If the scale is constrained to a power of two, the reconstruction is a bit shift, and a per-group shift of 4 bits fits in the same word as the data. Power-of-two scales cost resolution, since the scale can only be within a factor of two of the ideal, but they convert a floating-point multiply into an integer shift and they compress the metadata into a nibble. Formats that mix a small explicit scale with a power-of-two shift, as the OCP microscaling formats do at block sizes of 32, are taking this route: an 8-bit exponent shared by 32 elements, with the mantissa bits varying per element.

Calibration is per-group too

Choosing the scale per group means computing a minimum and maximum per group, and that statistic is estimated from data. With a group of 128 values, the sample maximum is a noisy estimator of the population maximum: it is the largest of 128 draws and depends heavily on which 128 draws were seen. A group whose calibration sample happened to contain an outlier gets a wide scale; the same group in a different sample does not. The result is a scale map that varies with the calibration set, and a model whose quantized weights differ between two runs that used different calibration batches.

The mitigation is to calibrate on enough data that each group’s extremes have stabilized, or to choose the range by a criterion that is less sensitive to the single largest value. Range calibration covers the criteria; the point here is that granularity and calibration interact, because a smaller group has fewer samples behind its range estimate.

There is also an interaction with the output-axis structure that is easy to miss. Per-channel scales for a layer followed by a batch normalization can be folded: the normalization’s per-channel affine transform composes with the weight scale, and the fused scale is the product. Per-group scales cannot be folded the same way, because the normalization’s parameters are per output channel while the group scales vary along the input axis. A fusion pass that assumes per-channel scales will either refuse to fold or fold incorrectly, and the failure appears as a small accuracy loss that no shape check catches.

When per-tensor is still right

Per-channel quantization is not universally better. It requires the runtime to know the axis, which means the format carries an axis convention and every kernel that reads the tensor must agree. It complicates concatenation, because two tensors concatenated along the channel axis have disjoint scale sets that must be unified or kept as separate segments. It complicates slicing, because a slice that does not respect channel boundaries produces a tensor whose scales no longer correspond to its rows.

For small matrices the metadata is a real fraction of the payload and the gain is small: a 128-by-64 matrix has 8192 weights and 128 scales, which is 6 percent overhead at 4 bits for a resolution improvement that may not be measurable. For activations computed per inference, a per-tensor scale is often kept because the range is estimated from a running statistic and the per-channel alternative would require a reduction across the batch that costs more than the precision it recovers.

The general rule that survives is: put the scale on the axis that the kernel already treats as a unit of work, and put the group boundary at the granularity the kernel’s tile already has. A scale that does not align with a tile forces the kernel to split work it wanted to do as a unit. A scale coarser than the tile leaves precision on the table for free.

There is a further wrinkle in the choice of group boundary relative to a cache line. A group of 128 four-bit weights occupies 64 bytes, which is a cache line on the common architectures, so a group boundary and a cache line boundary coincide. A group of 64 occupies 32 bytes and two groups share a line, which is fine for sequential access and awkward when a kernel wants to exchange the halves of a line between threads. Group sizes that are neither aligned to a line nor a clean divisor of one produce accesses that straddle lines at every group, and the penalty for that is larger than the metadata saving. The group size is therefore often chosen for the memory system first and for the distribution second, and the difference in resulting accuracy between the whole set of plausible sizes is usually smaller than the difference the kernel sees.

The quantization error introduced by any of these choices is not the whole story, because it propagates through subsequent layers and can be amplified or attenuated by the weights that follow. Which layers tolerate which granularity is the subject of layer sensitivity and bit allocation, and it is the reason a uniform per-channel 4-bit recipe applied to every layer is a starting point rather than an answer.