Weight-Only Quantization and the Memory-Bandwidth Bound

Reducing the precision of weights alone, leaving activations in floating point, is the least invasive form of quantization and the one most often used for large language models. It does not require the arithmetic to change: the weights are stored small and expanded to floating point on load, and the multiply runs in the same floating-point units as before. What it changes is the number of bytes that must cross the memory bus, and whether that helps depends entirely on whether the bus was the constraint.

The ratio that decides

A linear layer computing $y = Wx$ with $W$ of shape $[C_{\text{out}}, C_{\text{in}}]$ performs $2 C_{\text{out}} C_{\text{in}}$ floating-point operations and must read $C_{\text{out}} C_{\text{in}}$ weights. At $b$ bits per weight the byte count is $C_{\text{out}} C_{\text{in}} b / 8$. Arithmetic intensity, in operations per byte read, is

$$\frac{2 C_{\text{out}} C_{\text{in}}}{C_{\text{out}} C_{\text{in}} b/8} = \frac{16}{b}$$

which is 2 at 8 bits, 4 at 4 bits, 8 at 2 bits. Compare against the machine balance: a processor with a peak floating-point rate $F$ and a memory bandwidth $B$ saturates at $F/B$ operations per byte. A server accelerator with tens of teraflops of float32 throughput and a few terabytes per second of bandwidth sits at a balance of a few tens of operations per byte. A 4-bit weight-only kernel at an intensity of 4 is an order of magnitude below that, and is bound by memory, not by arithmetic.

The consequence is direct: the time to execute such a layer is the time to read its weights, divided by the achievable fraction of peak bandwidth. Reducing weight bits from 16 to 4 reduces that time by up to a factor of four, and no amount of arithmetic optimization improves it further. Conversely, an increase in arithmetic cost that fits under the bandwidth roof, such as the dequantization itself, is free.

Batch size changes the intensity. With $B_s$ sequences processed together, the same weight read serves $B_s$ inputs, so intensity becomes $16 B_s / b$. At 4 bits, a batch of 16 reaches an intensity of 64, which is at or above the balance on many devices. The layer becomes compute-bound and the bit-width reduction stops helping. This is why weight-only quantization is described as a decode optimization: during token generation the batch is the number of concurrently served sequences, often small, and during prefill the batch is the number of prompt tokens, which can be in the thousands.

What the kernel actually does

A weight-only kernel reads packed low-bit integers, expands each to a floating-point value, and multiplies. The expansion is where the design choices are.

The naive arrangement stores weights in a packed layout and, in the inner loop, extracts each value, converts it, and performs a fused multiply-add. Extraction from a 4-bit field costs a shift and a mask; conversion from integer to float costs another instruction; the multiply-add is one more. That is three instructions of overhead per multiply, and the resulting instruction count per byte read rises by a factor of four relative to a 16-bit kernel. If the kernel is memory-bound with a large margin, this does not matter. If it is near the balance, it does.

The alternative is a lookup table. Because a 4-bit value has 16 possible reconstructions, the 16 products $s \cdot (q - z) \cdot x$ for a given activation $x$ can be precomputed once per call, and the inner loop becomes a table lookup indexed by the packed nibble. The table costs 16 entries and is reused across the whole row. This trades arithmetic for a dependent load, and the load is from a small table that stays in the nearest cache level, so it is fast but not free. Implementations differ on which is better depending on the ratio of multiply throughput to load throughput on the target.

A third arrangement avoids per-element conversion entirely by keeping groups of weights in a packed form and using integer dot products against quantized activations, which is no longer weight-only: it is the integer path. Weight-only exists precisely to avoid that requirement.

// Expand one group of 4-bit weights and accumulate against a float activation row.
// w_packed holds two nibbles per byte, low nibble first; group_size weights per scale.
static void dequant_dot(const uint8_t *w_packed, const float *x,
                        float scale, int group_size, float *acc) {
    float sum = 0.0f;
    for (int i = 0; i < group_size; i += 2) {
        uint8_t byte = w_packed[i >> 1];
        int q0 = byte & 0x0F;          // low nibble
        int q1 = (byte >> 4) & 0x0F;   // high nibble
        sum += (float)(q0 - 8) * x[i];
        sum += (float)(q1 - 8) * x[i + 1];
    }
    *acc += sum * scale;
}

The loop above is the simplest correct form and shows where the cost sits: two extractions, two conversions, two multiply-adds per byte. A production kernel processes a tile of output rows at once so that each loaded activation element is reused across rows, which raises intensity without changing the weight bytes.

Why the packing layout matters more than the format

A 4-bit format with a group size of 128 has one scale per 64 bytes of packed weights. If the scale is stored immediately before its group, every group boundary interrupts the sequential read, and vectorized loads must be split. If the scales for an entire row are stored in a separate array, the weight read is contiguous and the scale read is a separate contiguous stream, which is friendlier to prefetching and to the memory controller.

The interleaved layout has one advantage: a single load brings the scale and its data together, so a kernel that processes groups independently needs one address computation. The split layout has the advantage that the scale stream is small and can be staged into shared memory once per tile, leaving the weight stream as the only large transfer. For long rows the split layout wins because the weight stream is what saturates the bus.

There is a related choice for the zero point. If the format is symmetric, the stored integer is signed and the reconstruction is a single multiply by the scale. If it is asymmetric, the zero point must be applied, and the kernel either subtracts it during expansion, which is an extra instruction per element, or folds it into a correction term computed from the row sum, as described in the affine decomposition. Folding is standard for integer kernels and less commonly done for weight-only kernels, where the extra term would itself be a floating-point reduction.

What the speedup does not cover

Weight-only quantization reduces the bytes read for weights. It does not reduce the bytes read for the KV cache, which during decode grows with context length and with batch size and quickly stops being a rounding error next to the weights. The arithmetic is worth doing explicitly. For a model with 32 layers, a hidden width of 4096, and full multi-head attention, one cached token occupies $32 \times 2 \times 4096$ elements across all layers, which is 262144 elements, or 512 KiB at 16 bits. A single sequence at a context of 8192 tokens therefore holds about 4 GiB, larger than the 4-bit weight set of a 7-billion-parameter model. Grouped-query attention reduces the figure by the ratio of key-value heads to attention heads, and that ratio is what decides whether the cache or the weights dominate.

Nor does it reduce the cost of the operations that are not matrix multiplications. Normalization, softmax, and the elementwise parts of attention run at whatever precision their inputs have, and they are a small fraction of the bytes but a nontrivial fraction of the launch overhead and the latency at small batch sizes. A layer-by-layer accounting is necessary before attributing a speedup to the weight format.

The third omission is accuracy. Weight-only quantization at 4 bits is generally close to the floating-point model on standard evaluations, but the error is not uniform across layers, and a single layer with an unfortunate weight distribution can dominate the degradation. The techniques that make 4-bit work reliably are not about the format; they are about choosing the rounding so that the layer’s output error is minimized, which is the subject of error-compensated rounding at the low end and of layer sensitivity for the assignment of bits across the model.

The boundary against the integer path

The alternative to weight-only is to quantize activations as well and run integer arithmetic. That path changes the intensity calculation only through the activation bytes, which are small compared with the weights, and it changes the arithmetic from floating-point multiply-add to integer multiply-accumulate. On hardware where integer throughput exceeds floating-point throughput by a factor of two or four, which is common on mobile and edge accelerators, the integer path wins on compute as well as on bytes. On server accelerators with large floating-point throughput and specialized low-precision matrix units, the comparison depends on whether the low-precision units are exposed to the kernel at all.

The choice therefore tracks the deployment target more than the model. Weight-only is the format that a general-purpose runtime can support without changing its arithmetic, and it is the one that produces a predictable bandwidth reduction on any hardware. The integer path produces a larger reduction on hardware designed for it and a smaller one elsewhere.

The remaining question for weight-only is where the bits are spent. Uniform 4 bits is a starting point; a model in which some layers are quantized to 8 and others to 4, chosen by measuring each layer’s sensitivity, extracts more quality at the same average width. That is the allocation problem, and it requires a way to measure the error each layer contributes, which is easier for weights than for activations because the weights do not change.