Inference Systems · Cluster
Quantization
Precision reduction and calibration.
- 01 Affine Quantization: Scale, Zero Point, and the Error They Introduce How a real-valued tensor is mapped onto a small integer grid, why the zero point exists, and where the rounding error lands inside a dot product.
- 02 Per-Channel and Group Scales: Granularity Against Metadata Cost Why one scale per tensor loses to one scale per output channel, how group-wise quantization spends extra bytes to buy resolution, and what the metadata costs at each granularity.
- 03 Range Calibration: Trading Clipping Error Against Rounding Error How the clipping threshold is chosen for a quantized tensor, why the minimum-error range is not the observed range, and how percentile and divergence criteria differ in what they preserve.
- 04 Weight-Only Quantization and the Memory-Bandwidth Bound Why 4-bit weights speed up token generation while leaving prefill almost untouched, how the dequantize-on-load kernel works, and where the arithmetic intensity ceiling sits.
- 05 Activation Outlier Channels and Mixed-Precision Decomposition Why a few hidden-state channels carry magnitudes far above the rest, how splitting those channels out of the integer path preserves accuracy, and what the split costs per token.
- 06 Requantization Inside the Kernel: From Int32 Accumulator to Int8 Output How a convolution kernel turns a 32-bit integer accumulator into the next layer's 8-bit input, why the multiplier is a fixed-point pair, and where saturation and rounding diverge from the reference.
- 07 Fake Quantization and the Straight-Through Estimator Why a rounding operation has no useful gradient, how training simulates quantization in the forward pass, and what the straight-through substitute does to the scale update.
- 08 KV Cache Quantization and Per-Token Scale Placement Why the key and value caches need different quantization axes, how per-token scales interact with the attention kernel, and what happens to the softmax when keys lose precision.
- 09 Layer Sensitivity and Bit Allocation Under an Average-Width Budget How per-layer sensitivity is measured, why a greedy allocation on a non-additive error metric misleads, and how to spend a fixed average bit width across a network.
- 10 Error-Compensated Rounding at Two and Three Bits How sequential rounding with a residual update turns a coarse grid into a usable one, why the column order matters, and when the compensation stops converging.