Noise Schedule Shape and the Signal-to-Noise Ratio Curve

A schedule is a curve, and the curve determines how much of the training signal comes from each noise level. Two models with identical architectures, identical data, and identical step counts can differ substantially in sample quality purely because one allocates its timesteps differently. The variable that governs this allocation is the signal-to-noise ratio, and the schedules in common use are best understood as different ways of distributing timesteps along it.

The practical question is not which schedule is correct but which region of the ratio curve the model needs to learn well. Coarse structure is decided at ratios near one and below; fine texture is decided at ratios well above one. A schedule that spends most of its steps in a region where the task is trivial wastes capacity, and a schedule that skips a region entirely produces samples with a characteristic defect.

Two conventions and what each one measures

The variance-preserving process keeps the total variance of the state at one throughout the chain by scaling the signal down as it adds noise. Its state at step t is a mixture whose signal variance is A_t and whose noise variance is 1 − A_t, so the ratio is

SNR(t) = A_t / (1 − A_t)

The variance-exploding process does not scale the signal. It adds noise of standard deviation σ_t to data of unit variance, giving

SNR(t) = 1 / σ_t²

The two conventions describe the same family of distributions up to a rescaling of the state, but they behave differently numerically. In the preserving case the state always has unit variance, so the network’s inputs are normalised by construction and the same architecture works at every timestep. In the exploding case the state’s variance grows with t, and the network must either be conditioned on σ_t explicitly or given inputs normalised by it. The preserving convention is easier to implement; the exploding convention is easier to analyse because its schedule is a single function of the noise scale with no interaction between the drift and the diffusion.

The identity that connects them is that a preserving process at time t is an exploding process with σ = √((1 − A_t)/A_t) applied to the rescaled state x/√(A_t). Any schedule can be translated into either convention, and the choice is a matter of which arithmetic is less error-prone in the implementation.

Why a linear β schedule under-spends at the ends

The original discrete formulation used β increasing linearly from about 1e-4 to about 0.02 over a thousand steps. The resulting A_t curve is not linear: because A_t is a product of (1 − β_s), its logarithm is approximately the negative sum of the β_s, so A_t ≈ exp(−Σ β_s). With β growing linearly, the sum grows quadratically, and A_t therefore decays like a Gaussian in t. The curve is flat near t = 0 and steepest in the middle.

Working the numbers on that schedule shows where the steps go. The ratio stays above 10 for the first 94 steps and falls below 0.01 at step 675. That leaves roughly a hundred steps at the clean end where the denoising task is nearly the identity, and roughly three hundred and twenty at the noisy end where the input carries little beyond global structure. The informative middle, where the ratio moves from 10 down to 0.01, is about 580 steps, and half of those sit below a ratio of one.

The waste is concentrated in the tail. At a ratio of 0.01 the signal contributes one percent of the variance, so the posterior over the clean sample is nearly the prior and the network’s best prediction is close to the data mean. Steps in that regime teach the marginal distribution, which the first few sampling steps already fix, and they cannot teach anything about a particular sample because the sample is no longer visible in the input. The clean-end steps have the opposite problem: the optimal prediction is nearly the identity, which is arithmetic the sampler could perform without a network.

The cosine schedule shifts the curve so that less of the chain sits in either dead zone. The same computation puts its ratio below 10 at step 189 and below 0.01 at step 937, so the tail below 0.01 shrinks from about 325 steps to about 63, and the informative middle occupies roughly three quarters of the chain instead of a little over half. The price is a longer stretch at high ratio, which is cheap because those steps are easy to learn, and a terminal ratio of about 10^-6 rather than 10^-5, which means the final state is closer to pure noise and the Gaussian starting point of the reverse process is a better match.

The cosine alternative

A schedule defined by A_t = cos²((t/T + s)/(1 + s) · π/2), with a small offset s that keeps A_0 slightly below one, produces a curve whose slope is roughly constant in the middle and whose endpoints are gentler than the linear schedule’s. The offset exists to prevent A_0 from being exactly one, which would make the first step noiseless and the model’s prediction at t = 0 degenerate.

The motivation is that the derivative of log SNR with respect to t is more nearly constant, so the model sees each octave of the ratio for a comparable number of steps. The comparison above shows the effect: the log ratio moves from about 5 to about −6 across the chain rather than from 3.5 to −9, so the same number of steps covers a narrower range of ratios at a more even rate. The trade is that the schedule ends at a lower ratio, which costs nothing, and starts at a higher one, which means the earliest steps are closer to noiseless and the finest detail is learned from fewer examples.

Which schedule wins depends on the resolution and the data. At low resolution the fine-detail region is small and the linear schedule’s waste at the top is cheap, so the difference is modest. At high resolution the fine-detail region is where most of the bits live, and a schedule that under-samples it produces samples that are structurally correct and texturally wrong.

import numpy as np

def linear_betas(T=1000, b0=1e-4, b1=0.02):
    return np.linspace(b0, b1, T)

def cosine_alpha_bar(T=1000, s=0.008):
    t = np.arange(T + 1) / T
    f = np.cos((t + s) / (1.0 + s) * np.pi / 2.0) ** 2
    a_bar = f / f[0]                  # normalise so that A_0 = 1
    betas = 1.0 - a_bar[1:] / a_bar[:-1]
    return np.clip(betas, 1e-8, 0.999)  # keep the step kernel valid

def snr(a_bar):
    return a_bar / (1.0 - a_bar)

The clipping in the second function is not cosmetic. The ratio A_(t+1)/A_t can exceed one by a rounding error at the flat start of the curve, which makes β negative and the transition kernel invalid. Clipping to a small positive floor keeps the chain monotone.

Where the ratio curve meets the data

The signal-to-noise ratio is not the only quantity that matters; it interacts with the scale of the data and the resolution of the samples.

At a fixed ratio, the absolute noise level depends on the signal variance. If the data are normalised to unit variance, a ratio of one means signal and noise contribute equally. If the data have variance well below one, the same ratio corresponds to a much smaller absolute noise, so the transition from structure to noise happens earlier in absolute terms and the schedule must be shifted to compensate. This is why the same schedule is not transferable between datasets without rescaling, and why implementations normalise the data range explicitly.

At higher resolution, the number of pixels per sample grows, and the reconstruction task at a given ratio becomes harder because the model must predict more independent values. The information content of a sample scales with the pixel count, so the ratio region where the model has the most to learn widens. A schedule tuned for 32×32 images allocates too few steps to the fine-detail region for 256×256 images, and the symptom is samples that look correct at a distance and noisy at full size.

The ratio curve also determines the behaviour of the sampler’s step sizes. In the probability-flow ODE, the velocity field’s magnitude is proportional to the score, and the score’s magnitude grows as the ratio increases because the distribution becomes more concentrated. A fixed number of uniform steps in t therefore takes large effective steps in the high-ratio region, where the field is stiff, and small ones in the low-ratio region. Solvers that adapt their step size to the local curvature handle this automatically; fixed-step solvers need a schedule that distributes steps according to the ratio rather than uniformly in t.

Choosing step counts for a given schedule

The step count and the schedule are not independent choices. A schedule that concentrates its ratio range into a narrow band of t needs more steps to resolve that band, while a schedule that spreads the range evenly can use fewer steps at the same accuracy. The relevant quantity is the number of steps per unit of log ratio, not the number of steps per unit of t.

For sampling, the reverse process is usually discretised uniformly in t because that is what the trained model expects: the network was trained on timesteps drawn from a particular distribution, and evaluating it at timesteps outside that distribution produces predictions that are extrapolations. Changing the sampler’s step count without changing the timestep values it queries is safe; changing the set of timesteps requires either retraining or interpolating the schedule consistently.

The interaction with guidance is worth noting. Guided sampling makes the field stiffer in the mid-ratio region, where the conditional and unconditional predictions differ most, so a schedule that is adequate at zero guidance can produce artefacts at high guidance. The usual adjustment is to reduce the guidance scale at the last few steps, which is equivalent to flattening the field in the region where the sampler is most sensitive.

Degenerate schedules

Three schedules fail in ways that are easy to diagnose.

A schedule whose β reaches 1 at any step makes that step destroy the sample entirely, and A_t becomes zero from that point on. Every later timestep then has zero signal, so the model receives no information and the reverse process has nothing to invert. The symptom is a sampler that produces the same output regardless of the starting noise.

A schedule whose β is too small throughout never reaches a terminal distribution close to normal, so the reverse process starts from a distribution the model has never seen. The samples inherit a bias toward the data’s mean, because the terminal state still contains residual signal that the model interprets as structure.

A schedule with a discontinuous jump in β produces a step where the marginal changes abruptly, and the Gaussian reverse kernel’s assumption that the transition is small is violated. The result is a visible seam in the trajectory, which appears in the output as a systematic artefact at a specific scale. Smooth schedules avoid this by construction, and the cosine form is smooth in both endpoints.

The likelihood accounting that depends on the schedule’s endpoint is taken up in bits per dimension for discretised data, where the terminal distribution’s mismatch with a standard normal enters the bound directly.