Layer Sensitivity and Bit Allocation Under an Average-Width Budget

Uniform bit width is the default because it is easy to specify and easy to implement. It is not the default because it is optimal. A network’s layers differ in how much quantization error they can absorb, by an order of magnitude in the extreme, and a deployment that must hit a memory or latency target measured in bytes is spending its budget evenly across layers that do not return value evenly.

The allocation problem is: given a target average width, assign a width to each quantizable tensor so that the resulting error is minimized. It is a knapsack problem with a non-additive objective, and both of those properties matter.

Measuring what a layer contributes

The first requirement is a per-layer sensitivity: a scalar saying how much the network’s output degrades when that layer alone is quantized to a given width. The direct measurement is to quantize one layer, run an evaluation set, and record the loss. That is one evaluation per layer per width, which for a network of a hundred layers and four candidate widths is four hundred evaluations, each requiring a forward pass over a dataset. It is feasible offline and it is what a careful allocation starts from.

The proxy measurement is cheaper and more informative. The error a layer introduces is the difference between its quantized output and its exact output, and that difference propagates through the rest of the network. The propagation is approximately linear for small perturbations, so the effect on the final loss is the layer’s output error projected through the Jacobian of the remaining network. The Jacobian is expensive; its diagonal approximation is not.

For a linear layer, the second-order analysis gives a compact form. If the layer’s output error is $\delta y$ and the loss is locally quadratic with Hessian $H$ with respect to the layer’s output, the loss increase is approximately $\tfrac{1}{2}\delta y^\top H \delta y$. The Hessian of the loss with respect to a linear layer’s weights is related to the second moment of its input, which can be estimated from a calibration set by accumulating $x x^\top$ over samples. That gives a per-weight importance score, and the layer’s sensitivity is a summary of those scores.

The trace of that matrix is the natural summary, and it has a concrete meaning: it is the expected squared norm of the layer’s input, which measures how much amplification a weight error receives. A layer whose inputs have large norm amplifies weight errors; a layer whose inputs are small does not. This is the reasoning behind Hessian-aware allocation, and it is why the input second moment is the quantity to collect.

The same measurement done for activations is less reliable, because the activation distribution depends on the input and the second moment estimated from a calibration set is an estimate of a quantity that varies. Weight sensitivity is stable; activation sensitivity is not, and an allocation that treats them symmetrically will over-trust the activation estimates.

Why the objective is not additive

If each layer’s error contributed independently to the total, the allocation would be a standard knapsack: sort layers by the error reduction per bit saved, and take them in order. The independence assumption is wrong in a specific and consequential way.

Quantization errors in successive layers compose. A layer’s output error enters the next layer as an input error, and the next layer’s weights multiply it. If the next layer’s weights have large singular values, the error is amplified; if they contract, it is attenuated. The composition means that quantizing layer $i$ and layer $i+1$ together does not produce the sum of their individual errors, and in a network with residual connections the errors from different branches add in the residual stream and can partially cancel.

There is a second non-additivity that runs the other way. Some layers are redundant: their output contribution is small relative to the residual, so an error in them is attenuated by the residual sum. Quantizing such a layer to a very low width costs little, and the measurement of its sensitivity already reflects that. The interaction appears when several redundant layers are quantized at once: their errors, individually attenuated, may be correlated and add rather than cancel.

The practical consequence is that a greedy allocation using individually measured sensitivities is a heuristic with a known failure mode, and the failure mode is a network that validates well layer-by-layer but degrades when the whole allocation is applied. The check is to evaluate the full allocation, not to trust the sum of its parts.

Search over the assignment

The space of assignments is exponential in the number of layers. For a hundred tensors and four widths there are $4^{100}$ assignments, so the search must be restricted.

The standard restriction is a Lagrangian relaxation. Instead of minimizing error subject to an average-width constraint, minimize error plus a penalty times the total width, and vary the penalty. For a fixed penalty, each layer can be optimized independently, which is a small search per layer. Varying the penalty traces out the trade-off curve, and the assignment that hits the target width is read off the curve. The relaxation is exact only when the objective is convex and separable, which it is not, but the resulting assignments are good starting points.

A second restriction is to search over a small number of candidate widths per layer rather than all widths. In practice the useful set is three or four values, and the allocation is a choice among them. With that restriction, the number of combinations is still large but a beam search or a coordinate descent converges in a modest number of full evaluations.

The third approach removes the search by making the width differentiable. Each layer’s width is relaxed to a continuous value, the quantization error is modelled as a function of that value, and gradient descent on the relaxed problem produces a distribution over widths that is then discretized. The model of error versus width is the weak point: it is usually an assumed form, such as error proportional to $2^{-2b}$, and the assumption is wrong for layers whose error is dominated by clipping rather than rounding. When the model is wrong, the allocation follows the model rather than the data.

def allocate(sensitivities, widths, target_avg, n_layers):
    """Greedy: start every layer at the widest setting, then demote the
    layer whose demotion costs the least error per bit saved."""
    cur = [max(widths)] * n_layers
    total_bits = sum(cur)
    while total_bits - 1 >= target_avg * n_layers:
        best = None
        for i in range(n_layers):
            j = widths.index(cur[i])
            if j == 0:
                continue
            nxt = widths[j - 1]
            cost = sensitivities[i][nxt] - sensitivities[i][cur[i]]
            saving = cur[i] - nxt
            score = cost / saving
            if best is None or score < best[0]:
                best = (score, i, nxt)
        if best is None:
            break
        _, i, nxt = best
        total_bits -= cur[i] - nxt
        cur[i] = nxt
    return cur

The greedy loop above is the additive heuristic, and it is worth having in mind precisely because it is the thing the non-additivity defeats. It is a reasonable first pass and a poor final answer.

Where the budget actually binds

An average-width target is not the constraint that a deployment usually faces. The constraints are a memory ceiling, a latency ceiling, and sometimes a per-kernel throughput requirement, and these bind differently.

Memory is the easiest to convert: the bytes are the sum of the tensor sizes times their widths plus metadata, so an average width translates directly. But memory is allocated per tensor, and a tensor’s size is fixed, so the allocation is a true knapsack in bytes and the “average width” is a derived quantity rather than a control.

Latency is harder. A layer at 8 bits and a layer at 4 bits may take the same time if the 4-bit kernel does not have a faster path, in which case the allocation improves memory without improving latency. Conversely, a layer at a width the hardware does not support natively may run through a slower fallback and cost more than its neighbour at a higher width with a native kernel. The set of widths worth considering is therefore the set the hardware implements efficiently, not the set that is mathematically available.

Throughput adds a third dimension: on hardware with a low-precision matrix unit, the unit’s shape constrains the tile sizes, and a layer whose dimensions do not tile well at a given width may be slower at 4 bits than at 8. The allocation problem on such hardware includes a per-layer performance model that is a property of the kernel library, not of the arithmetic.

The sensitivity that allocation usually misses

Per-layer sensitivity as measured above treats the layer’s weights as the source of error. For a transformer, the more sensitive quantity is often the activation, and specifically the outliers discussed in the mixed-precision decomposition. A layer with a severe outlier channel has an activation quantization error that is not captured by a weight-Hessian analysis, because the error comes from the input side.

There is also a structural asymmetry: the first and last layers of a network are usually kept at higher precision, and this is not because their measured sensitivity is highest. The first layer sees the raw input, whose distribution is the least controlled and the least like the calibration set; the last layer produces the output, whose errors are not attenuated by anything downstream. Both arguments are about the absence of a downstream averaging mechanism, not about the Hessian, and they are the reason the practical rule “keep the edges wide” survives even when a measured allocation suggests otherwise.

A third consideration is that sensitivity measured at one width does not extrapolate. A layer that is insensitive at 8 bits may be catastrophic at 2, because the error mechanism changes: at 8 bits the error is rounding, which is small and roughly independent across elements; at 2 bits the error is dominated by the coarse grid and by clipping, and a layer whose weights have a heavy-tailed distribution crosses from “fine” to “destroyed” between those widths rather than degrading smoothly. An allocation that interpolates sensitivity across widths from measurements at the endpoints will misjudge the middle.

What the allocation is worth

The gain over uniform width depends on the heterogeneity of the network. A network whose layers have similar sensitivity gains little, and the bookkeeping cost of mixed widths, which includes per-tensor metadata, kernel dispatch on width, and a checkpoint format that records the assignment, may exceed the benefit. A network with a few clearly dominant layers gains substantially, because those layers can be kept at 8 bits while the bulk drops to 4 or below.

The interaction with calibration is that the allocation and the range choice are coupled: the range for a layer at 4 bits should be chosen by a criterion that accounts for the coarser grid, and a range optimized for 8 bits is not the right range for 4. Running the allocation with ranges fixed from an 8-bit calibration, then re-quantizing the selected widths with their own calibration, is the practical sequence, and it means the allocation is approximate. The approximation is acceptable because the sensitivity ordering is usually stable across reasonable range choices, even when the absolute errors are not.

The deepest limitation is that the whole framework assumes the error is a scalar that can be minimized. What actually matters is task accuracy, which is a nonlinear function of the error and is not monotone in it for every input. A layer whose error changes the argmax of a classification is more damaging than one whose error shifts a logit by the same amount, and no Hessian captures that distinction. Allocation is therefore a way to spend a budget sensibly, not a way to guarantee an outcome, and the final check is always an evaluation of the assembled model rather than the sum of its measured parts.