Forward Diffusion as Closed-Form Gaussian Corruption

A diffusion model is trained by destroying data in a controlled way and learning to reverse the destruction. The destroying half is fixed in advance and contains no learnable parameters. Its design is where most of the practical differences between diffusion variants originate, because the choice of how noise is added determines the shape of the learning problem, the numerical conditioning of training, and the number of sampling steps required at generation time.

The property that makes the whole scheme work is that although the corruption is defined as a long chain of small steps, the chain can be collapsed: the distribution of the corrupted sample at any step, conditioned on the clean sample, is Gaussian with a closed-form mean and variance. No iteration is needed to compute it. This single fact is what makes training a diffusion model as cheap as training a regression model.

The chain and its one-step kernel

Let x₀ be a clean sample and define a sequence x₁, …, x_T by repeatedly adding a small amount of Gaussian noise. Each step is

x_t = √(1 − β_t) · x_{t−1} + √(β_t) · ε,    ε ~ N(0, I)

with a schedule of small positive coefficients β_t. The coefficient on the previous sample is written as a square root so that the variance of the result is preserved when the input has unit variance: (1 − β) + β = 1. This variance-preserving convention is a choice, not a requirement, and the alternative convention scales the noise up while leaving the signal intact.

The step is affine in the previous sample plus independent Gaussian noise, so the transition kernel is Gaussian: q(x_t | x_{t−1}) = N(√(1 − β_t) x_{t−1}, β_t I). Composition of affine Gaussian maps is affine Gaussian, which is what allows the chain to be collapsed.

Collapsing the chain into one equation

Define α_t = 1 − β_t and the cumulative product A_t = α₁α₂…α_t. Applying the one-step kernel t times gives

x_t = √(A_t) · x₀ + √(1 − A_t) · ε,    ε ~ N(0, I)

so that q(x_t | x₀) = N(√(A_t) x₀, (1 − A_t) I). The derivation is induction: if x_(t−1) = √(A_(t−1)) x₀ + √(1 − A_(t−1)) ε′, then multiplying by √(α_t) and adding √(β_t) ε gives a mean of √(α_t A_(t−1)) x₀ = √(A_t) x₀ and a variance of α_t (1 − A_(t−1)) + β_t = 1 − A_t, because α_t A_(t−1) = A_t and α_t + β_t = 1.

The consequence is that any training example can be corrupted to any timestep in one operation, with no loop and no accumulated floating-point error. A_t is a scalar that depends only on the schedule, so it can be precomputed as a table of T values and looked up by index. Training then consists of drawing a timestep uniformly, drawing noise, forming the corrupted sample, and asking the network to predict something about it.

import numpy as np

def schedule(T=1000, beta_start=1e-4, beta_end=0.02):
    betas = np.linspace(beta_start, beta_end, T)      # linear schedule
    alphas = 1.0 - betas
    alpha_bar = np.cumprod(alphas)                   # A_t, monotonically decreasing
    return betas, alphas, alpha_bar

def corrupt(x0, t, alpha_bar, rng):
    # t is an integer index; alpha_bar[t] is A_t
    a = alpha_bar[t]
    eps = rng.standard_normal(x0.shape)
    return np.sqrt(a) * x0 + np.sqrt(1.0 - a) * eps, eps

The array alpha_bar is the entire forward process in T floating-point numbers. At T = 1000 with float32 storage it occupies 4 KB, which is why every implementation caches it rather than recomputing products at each step.

Why A_t is the only schedule variable that matters

Every quantity of interest in the forward process is a function of A_t alone. The signal-to-noise ratio of the corrupted sample is A_t / (1 − A_t), a monotone decreasing function that crosses 1 at the point where the signal and noise variances are equal. The fraction of the original signal remaining is √(A_t), and the fraction of variance that is noise is 1 − A_t.

This reduction is what makes schedule design tractable: a schedule is a curve from A_t near 1 at t = 0 to A_t near 0 at t = T, and the shape of that curve in between is the design decision. A linear β schedule with β from 1e-4 to 0.02 produces a curve where A_t decays slowly at first and then rapidly, so most of the noise is added in the middle of the chain. The signal is still largely intact for the first few hundred steps, which means those steps are easy to denoise and contribute little to the learning signal.

The endpoint matters for the opposite reason. If A_T is not close to zero, the final step still has residual signal, and sampling from a standard normal at the start of the reverse process is a mismatch with what the model was trained to expect. A common linear schedule reaches A_T of a few times 10^-5 at T = 1000, which is small enough that the residual signal is negligible relative to unit noise. Schedules that stop earlier leave a visible bias in generated samples.

The reverse direction is also Gaussian, given the clean sample

The forward chain is not reversible in general, but it is reversible when conditioned on x₀. Bayes’ rule applied to the Gaussian kernel gives

q(x_(t−1) | x_t, x₀) = N(mean_t(x_t, x₀), var_t I)

with a mean that is a weighted combination of x_t and x₀, and a variance that depends only on the schedule and equals β_t (1 − A_(t−1)) / (1 − A_t). The posterior mean can be rearranged to express x₀ in terms of x_t and the noise that produced it:

mean_t = (1/√α_t) · ( x_t − (β_t / √(1 − A_t)) · ε )

which is the form that motivates predicting the noise rather than the clean sample. The network is asked for ε, and the mean of the reverse step follows by arithmetic.

The variance of the reverse step is not learnable in the original formulation, and this is a significant constraint: the reverse process is assumed to have the same functional form as the forward one, which forces the generative model to be a Gaussian chain. Later work learns the variance as well, which helps at small step counts because the fixed variance is too large when steps are coarse. The cost is an extra output channel and a more delicate training objective.

What the fixed process does and does not constrain

The forward process has no parameters, so it cannot overfit and it cannot be trained. It is a data transformation, and every property of the generative model comes from the network that learns the reverse. This separation is the main structural difference from the variational autoencoder, where the encoder is learned and can collapse.

The absence of a learned encoder has a concrete cost at sampling time. Generating a sample requires iterating the reverse chain T times, each with a full forward pass of the network. With T = 1000 that is a thousand network evaluations per sample, against one for a single-pass generator. This is the reason diffusion models were initially impractical and the reason so much subsequent work is about reducing the number of steps without changing the learned distribution.

The fixed process also determines the dimensionality of the learning problem. The network sees inputs at every noise level, from nearly clean to nearly pure noise, and must handle all of them with one set of weights. The signal-to-noise ratio varies by orders of magnitude across the chain, so the regression target has a correspondingly wide dynamic range. Normalising the loss by the noise level, or weighting timesteps by their contribution, is the standard response.

There is a symmetry worth noting: the forward process can be run in reverse from noise only if the score of the data distribution is known, which is what the network learns. The relationship between the noise-prediction objective and the score is developed in denoising score matching, and the choice of how the noise scale is distributed across the chain is taken up in noise schedule design.

Degenerate cases and boundary behaviour

Several failure modes follow from the arithmetic rather than from training.

If A_t underflows to zero in float32, the corrupted sample becomes pure noise for all later timesteps and the network receives no gradient from them. With a thousand-step linear schedule A_T is a few times 10^-5, which is representable, but longer chains or more aggressive schedules can reach the denormal range. Computing the schedule in double precision and casting the table to float32 avoids the underflow, because the table is small.

If β_t is too large at the start, the first few steps destroy most of the signal, and the model has almost no near-clean examples to learn from. The reverse process then has to make large jumps at low noise, where its Gaussian assumption is poorest. Schedules that start small and grow keep the early steps gentle, which is why the linear schedule starts with a β two orders of magnitude below its final value.

If the data are not normalised to unit variance, the variance-preserving identity no longer holds and A_t does not correspond to the signal fraction. Images scaled to [0, 1] have a variance far below 1, so the noise dominates much earlier than the schedule suggests. The usual fix is to rescale to [−1, 1], which puts the variance near 1 for saturated images, or to accept the mismatch and let the network learn the effective scale.

The final step of the reverse process is often given zero variance, on the argument that the last transition should land exactly on the clean sample rather than on a noisy version of it. This is a modelling choice that removes one source of sampling noise and slightly changes the implied distribution; the difference is small at large T and noticeable at small T.