Cosine and Linear Decay Schedules as Step-Size Integrals

A decay schedule is usually described by its shape and its endpoints: “cosine from $10^{-3}$ to zero over the run”. That description underspecifies what the schedule does, because the quantity that determines where the parameters end up is not the rate at any single step but the integral of the rate over the run. Two schedules with the same peak and the same total number of steps but different shapes distribute the same total displacement differently across the phases of training, and it is the distribution, not the peak, that changes which region of parameter space the run settles in.

Making that concrete requires one accounting identity and one observation about gradient noise.

Displacement is the sum of step sizes

For plain gradient descent with a schedule $\eta_t$, the total distance travelled in parameter space is bounded by $\sum_t \eta_t |g_t|$, and for a run whose gradient magnitude is roughly constant over a phase, the distance travelled in that phase is proportional to the sum of the rates in it. Writing $T$ for the total number of steps and $S = \sum_{t=1}^{T}\eta_t$, the schedule’s content is the fraction of $S$ allocated before any given step.

Three standard shapes, all starting from $\eta_0$ and ending at zero:

• Linear: $\eta_t = \eta_0 (1 - t/T)$, giving $S = \eta_0 T/2$.
• Cosine: $\eta_t = \frac{\eta_0}{2}(1 + \cos(\pi t/T))$, giving $S = \eta_0 T/2$ as well, since the cosine over a half-period has mean one half.
• Inverse square root: $\eta_t = \eta_0/\sqrt{1 + t/\tau}$, which does not reach zero and has $S \approx 2\eta_0\sqrt{\tau T}$ when $T \gg \tau$.

The first two have identical area. Cosine spends more of the budget in the middle and less at the ends: its derivative is zero at $t=0$ and $t=T$, so the rate stays near its peak for the first part of the run and approaches zero slowly at the end. Linear spends equally per step at first and then drops uniformly. Their difference at any given step is bounded by roughly half the peak rate, and for the same endpoints, that difference integrates to zero.

The practical consequence is that swapping linear for cosine at the same peak and same length changes the trajectory but not the budget. The reports that one is better than the other are therefore about the shape’s effect — the timing of the phase where the rate is high relative to the phase where the model is refining — not about total displacement.

Why the end of the schedule matters more than the middle

Near a minimum, the gradient is small but the noise in the gradient is not. For a batch of size $B$ with per-example gradient variance $\sigma^2$, the stochastic gradient’s variance is $\sigma^2/B$, independent of how close the parameters are to the minimum. The iterates perform a random walk whose stationary radius scales as $\eta \sigma/\sqrt{B}$ for constant step size, so the final loss depends on the rate at the end in a way it does not depend on the rate in the middle.

This is the mechanism behind decay in general: the rate must fall to reduce the stationary distribution’s width. A schedule that holds the rate high until the last step reaches a neighbourhood of the minimum and then jitters within it at a radius set by the high rate. A schedule that spends its last fraction of steps at very small rates continues to move, but slowly, and the jitter shrinks. Since the loss’s dependence on the iterate is quadratic near a minimum, halving the stationary radius quarters the excess loss above the minimum.

Cosine’s zero derivative at the endpoint is what makes the final phase quiet: the last $10%$ of steps receives substantially less than $10%$ of the total budget, and the rate at the final step is zero. Linear’s constant slope reaches the same endpoint but spends more of the final phase at larger rates.

import math

def linear(eta0, T):
    return lambda t: eta0 * max(0.0, 1.0 - t / T)

def cosine(eta0, T):
    # half-cosine; reaches exactly zero at t == T
    return lambda t: 0.5 * eta0 * (1.0 + math.cos(math.pi * min(1.0, t / T)))

for name, sched in (("linear", linear(1e-3, 10000)),
                    ("cosine", cosine(1e-3, 10000))):
    area = sum(sched(t) for t in range(10000))
    print(name, round(area, 4))

Both areas are approximately the same value, which is the point: the identity holds for any half-cosine paired with the matching linear ramp.

Restarts and the cost of re-heating

A warm restart raises the rate back to its peak at a scheduled step and decays again. The mechanism is that a raised rate ejects the iterate from the current basin’s neighbourhood and lets it explore, on the theory that a lower-loss basin exists nearby. The cost is that each restart discards the refinement achieved by the previous cycle’s tail: the noise floor rises again, and the steps immediately after the restart move the parameters at a distance set by the peak rate.

The accounting is straightforward. With $k$ cycles of equal length $T/k$, each cycle contributes $\eta_0 T/(2k)$ to the total displacement, so the total is unchanged, but the number of quiet tail phases is multiplied by $k$ and the number of noisy high-rate phases is multiplied by $k$ as well. Whether this helps depends on whether the loss surface has multiple basins that the high rate can reach and the low rate can settle in; where it does not, restarts spend a fraction of the budget on re-exploration.

A refinement that changes the accounting is to shorten each cycle geometrically, so later cycles spend less time at high rates. The total remains a convergent sum, and the final cycle ends with a long quiet tail. The schedules used in practice at large scale are usually a single decay rather than a sequence of cycles, because a single decay allocates the largest quiet tail.

Coupling to batch size

The gradient’s variance scales as $1/B$, so the noise floor’s radius $\eta\sigma/\sqrt{B}$ depends on the product $\eta/\sqrt{B}$ rather than on $\eta$ alone. Doubling the batch lowers the noise radius by $\sqrt{2}$, which is equivalent to halving the rate, except that the number of steps per epoch is also halved. This is the content of the equivalence between decaying the rate and increasing the batch: a schedule with $T$ steps at rate $\eta$ and batch $B$ has, per unit of data seen, the same trajectory as one with $T/2$ steps at rate $\eta/2$ and batch $2B$ only if the gradient’s mean is not changing. Since the mean gradient does change over the run — that is what training is — the equivalence is approximate and holds best in the late phase where the mean is small and the noise dominates.

For a fixed total compute budget in example-passes, the number of parameter updates is $T = \text{examples}/B$, and the schedule’s total displacement is $\eta_0 T/2 = \eta_0 \cdot \text{examples}/(2B)$. Increasing $B$ for fixed compute decreases displacement unless $\eta_0$ is raised, which is why large-batch recipes raise the peak rate and shorten the schedule together. The interaction with warmup length is the subject of the early instability window: a higher peak requires a longer ramp relative to the same number of steps.

Schedules that do not decay to zero

Not every schedule ends at zero. Constant rate after warmup, and a final phase at a small constant rate, are both used. The reason is that a zero final rate makes the parameter’s movement vanish entirely, which is fine for convergence but means that any transient effect — a batch of unusual data, a numerical glitch — that occurred at the end of the decay is frozen into the weights. A small constant floor allows the last part of training to correct such effects.

The cost is the noise floor: with rate $\eta_f$ at the end, the stationary radius is $\eta_f\sigma/\sqrt{B}$, and the final loss includes an excess proportional to its square. Choosing between a floor and zero is choosing between a bounded correction capability and a bounded excess loss. Neither choice interacts with the shape of the preceding decay except through the total budget.

A note on the peak

The peak rate is the schedule’s most consequential parameter and the least determined by the shape analysis. Its upper bound is set by the stability of the largest curvature direction: for a quadratic with curvature $L$, descent converges only for $\eta < 2/L$, and a stochastic method with noisy gradients needs the rate small enough that the noise radius is tolerable, which is a stricter condition. Its lower bound is set by the budget: a peak too low does not traverse enough of the loss surface within $T$ steps. The decay shape sits between these bounds and changes how the fixed budget is spent, which is why a shape change is usually worth far less than a factor of two in the peak, and why the schedules used across models from different families differ in shape while their peak rates differ in order of magnitude.