Gradient Clipping by Norm and Loss-Scale Rescaling

Gradient clipping replaces the gradient vector $g$ with $g \cdot \min(1, c/|g|)$ for a threshold $c$. The operation is a projection onto a ball of radius $c$: vectors shorter than $c$ pass through, longer ones are scaled down to length exactly $c$. It changes the direction not at all, which is the property that distinguishes it from any form of gradient normalisation, and it changes the magnitude only when the magnitude exceeds a bound the practitioner declares acceptable.

The reason to do this is that the loss is not globally Lipschitz in the parameters, and no local argument based on the gradient at the current point says anything about the next point when the next point is far away. A step of size $\eta |g|$ can cross a region where the loss is enormous, and the gradient there, evaluated next step, is enormous too. Clipping bounds the distance travelled per step by $\eta c$ regardless of how large the current gradient is, which converts an unstable feedback loop into a bounded process. The question is what the bound should be, how the norm is aggregated across parameters, and how the operation behaves when the gradient is represented in reduced precision.

The norm that is clipped

Implementations compute a single scalar over a group of tensors rather than clipping each tensor separately. The reason is that separate clipping changes the relative scale between parameters: a tensor with one large entry would be scaled down by a large factor while another tensor’s gradient passes unchanged, so the update is no longer a scaled version of the true gradient and the descent direction is distorted. A global norm preserves the direction of the concatenation of all gradients in the group.

For a group with tensors $g^{(1)}, \dots, g^{(k)}$, the norm is $\sqrt{\sum_i |g^{(i)}|_2^2}$, computed by accumulating the sum of squares over all elements. The cost is one pass over the parameter gradients, and it must be done before any tensor is scaled, which forces a synchronisation point in a data-parallel setting: the norm depends on every parameter’s gradient, so the scaling factor cannot be computed locally on one device. This is the structural cost of global clipping in distributed training — an all-reduce of one scalar in addition to the all-reduce of the gradients themselves.

import math

def clip_global_norm(tensors, max_norm):
    total = 0.0
    for t in tensors:
        for v in t:
            total += v * v
    norm = math.sqrt(total)
    if norm <= max_norm:
        return 1.0
    scale = max_norm / (norm + 1e-12)
    for t in tensors:
        for i in range(len(t)):
            t[i] *= scale
    return scale

The 1e-12 in the denominator guards the case where the norm is exactly zero; dividing by a zero norm produces a NaN that propagates into the parameters, and a zero gradient is entirely plausible in a group of parameters that a mask has excluded. The scale factor returned is itself a useful diagnostic: a value well below one indicates that the threshold is binding.

Per-parameter-group clipping, using separate thresholds for embeddings and for the rest, is a compromise: it preserves direction within each group and fixes the relative scale between groups by the ratio of thresholds. It costs per-group norms and does not have a single scalar to report.

Choosing a threshold

The threshold $c$ has units of the gradient norm and scales with the model’s size, so a value transferred between architectures is meaningless without rescaling. Two principled anchors exist.

The first is the size of the gradient at initialization under the variance-preserving scheme. If the parameters are drawn so that each layer preserves the variance of the signal and of the backpropagated gradient, then at step zero the gradient’s per-parameter magnitude is set by the loss’s curvature with respect to those parameters, and the total norm scales as the square root of the parameter count for a fixed per-parameter scale. A threshold of order the square root of the parameter count is therefore not a sensitive choice; thresholds that are constant across model sizes effectively clip more strongly for larger models.

The second anchor is the reason clipping was introduced for recurrent networks: the explosion of the product of Jacobians along a sequence. The bound on the gradient of a recurrent loss grows with the sequence length; clipping at a length-independent threshold is what keeps the update bounded as the sequence grows. The original analysis chooses the threshold as a fraction of the norm that the parameters can tolerate being moved, not from a statistical property of the gradient.

In practice the threshold is usually set between $10^{-1}$ and $10$ for language models, and the reported quantity is the fraction of steps at which clipping binds. A fraction near zero means the threshold never matters and can be raised; a fraction near one means the optimizer is effectively running at a fixed step size $c$ and the learning rate’s role has changed.

Interaction with loss scaling in reduced precision

Training in reduced-precision floating point forces a combination of two scale factors that are easy to confuse. For the IEEE 754 binary16 format, the largest finite value is 65504 and the smallest positive subnormal is about $6\times10^{-8}$, which gives a dynamic range of roughly nine decimal orders of magnitude and a smallest normal value near $6\times10^{-5}$. Gradients in a deep network are frequently smaller than the smallest normal value, so they are represented as subnormals or flushed to zero. To avoid that, the loss is multiplied by a large constant $S$ before the backward pass, which multiplies every gradient by $S$; the resulting gradients are divided by $S$ before the optimizer sees them.

The consequence for clipping is that the threshold applies to the scaled gradient if clipping happens before the unscale, which is the natural order because the unscale is a separate step. A threshold chosen for real gradients will then bind at every step. The correct sequence is: compute the backward pass with the scaled loss, unscale the gradients by dividing by $S$, then clip, then step. Frameworks that fuse the unscale into the optimizer’s step, and expose clipping as a separate operation, require the caller to know which order their version implements.

The second interaction is the dynamic loss scale itself. When a step produces non-finite gradients — the overflow condition — the scale is reduced and the step is skipped. A gradient that overflows has already lost its information: the infinity in one element makes the global squared norm infinite, so the clip would produce a scale factor of zero and the step would be a no-op with a zero gradient. The overflow check must therefore precede clipping, and a careful implementation treats a non-finite norm as a signal to skip the update rather than to clip.

A subtlety: clipping large gradients is not a substitute for choosing a loss scale that keeps them representable. Clipping after the unscale cannot repair a gradient that was already flushed to zero in the forward precision — the information is gone before the clip is applied. The problem is in the representation range, not in the magnitude.

Clipping against the adaptive denominator

An adaptive optimizer divides by a root-mean-square estimate of the gradient. If clipping is applied before the optimizer, the clipped gradient enters the second-moment estimate, so a period of sustained clipping reduces the estimate’s magnitude and the effective step size for those coordinates. The behaviour is a feedback loop: clipping binds, the denominator falls, the ratio rises, and once the raw gradient falls below the threshold the update is larger than it would have been without clipping. The analysis of the interaction in decoupled weight decay runs in the opposite direction but the mechanism is the same — a term applied outside the gradient changes what the optimizer’s statistics measure.

The alternative is to clip the update after the adaptive division rather than the gradient before it. That bounds the step directly and leaves the statistics pure, but the direction can then be dominated by noise in coordinates with small denominators, since the scaling by one factor preserves the relative noise amplification. Both designs are used; they bound different quantities.

What clipping cannot fix

Clipping bounds the size of the step but not the damage a large step does within one iteration. If a single step of norm $\eta c$ moves the parameters to a region where the loss is defined but the model is degenerate — all units saturated, all attention weights collapsed to one position — the following gradients are small and clipping does nothing, because the problem is not magnitude. The mechanism protects against the explosive case only.

A related limitation is that the clip’s threshold is in gradient space, and gradient norms vary by orders of magnitude between the early and late phases of training as the loss decreases. A threshold tuned to bind occasionally at the start is inert later; a threshold tuned for the end will bind constantly at the start. Reporting the clip fraction per logging interval, and relating it to the learning rate schedule of warmup and the early instability window, is the only way to know which regime is in force.