Bits per Dimension and Likelihood on Discretised Data

Images are stored as integers. A model of images is almost always a continuous density, because continuous densities are differentiable and integer-valued ones are not. The mismatch between the two is not a detail: it changes the likelihood by a factor that depends on the bit depth, it makes the reported number sensitive to a dequantisation choice, and it produces comparisons between models that are measuring different things.

Bits per dimension is the standard normalisation used to compare models across resolutions and channel counts. Understanding what it divides by, and which quantity in the model it is computed from, is the difference between a number that is comparable and one that is not.

The quantisation gap

An 8-bit image has 256 levels per channel. The continuous density p(x) assigns density to every real value, and the probability of the discrete value k is the integral of that density over the interval assigned to k:

P(k) = ∫_{k/255 − 1/510}^{k/255 + 1/510} p(x) dx

where the image values are mapped to [0, 1] and each level occupies a bin of width 1/255 centred on its value. If the density is roughly constant across a bin, the integral is approximately p(k/255) · (1/255), so the log probability of the discrete value is the log density plus the log of the bin width.

That additive term is the whole of the issue. A model evaluated as a continuous density reports log p(x) with units of nats per continuous sample; converting to a discrete probability requires adding D · log(1/255) for a sample with D scalar values, where D = H·W·C. For a 32×32 RGB image, D is 3072, and the correction is a large negative number. A model that reports a good continuous log likelihood without the correction appears to assign far more probability to real images than any discrete model could, because the continuous density can exceed one and does so wherever the data are concentrated.

The reverse mistake is equally common. A model trained on continuous data with a Gaussian observation model has a likelihood that depends on the assumed variance, and comparing it against a discrete model’s cross-entropy requires matching the bin width to that variance. When the variance is set to one and the data are in [0, 1], the implied bin width is the unit interval, and the comparison is off by a factor of 255 per dimension, which is about eight bits per scalar value.

Defining bits per dimension precisely

The usual definition is

bpd = − log₂ P(x) / D

where P(x) is the probability the model assigns to the discrete sample and D is the number of scalar values, counting channels. For an RGB image at H×W, D = 3HW; for a grayscale image, D = HW. The normalisation by D makes models at different resolutions comparable in the sense that a value of 1.0 bpd means the model spends one bit per scalar value, and a uniform model over 256 levels spends exactly 8.0.

The choice of D rather than the number of samples or the number of pixels is what makes the number resolution-independent in the useful sense. Doubling the linear resolution quadruples D, so a model that assigns the same total probability to a larger image reports a lower bpd, which reflects that each additional value is more predictable given the others. Whether that is the right normalisation depends on the question: for compression, bpd is the right unit because it measures the code length per value; for comparing architectures at different resolutions, it is not, because the intrinsic predictability of a value depends on how many neighbours it has.

The correction to continuous likelihoods is applied per dimension, so it does not change with resolution. A model reporting a continuous log likelihood in nats is converted by

bpd = ( −log_likelihood_nats / ln 2 + D · log₂(255) ) / D

which simplifies to the nats-per-dimension divided by ln 2, plus log₂(255) ≈ 7.994. The constant is close to eight but not exactly eight, because the bin width is 1/255 rather than 1/256. Using 8.0 instead introduces a systematic offset of about 0.006 bits per dimension, which is small but larger than the differences between competitive models on standard benchmarks, so it matters for a leaderboard and not for anything else.

Dequantisation as a modelling choice

Training a continuous model on discrete data requires either a discrete observation model or a dequantisation step that adds uniform noise to the integer values before training. The uniform noise makes the training distribution continuous and matches, in expectation, the uniform density within each bin. A model trained this way can be evaluated by integrating its density over each bin, which for a Gaussian mixture is a sum of error-function differences and is exact.

The alternative is to train the continuous model on the quantised values without noise, treating them as points. This is simpler and it is wrong in a specific way: the model learns a density with modes at the integer values and near-zero density between them, so its density at the quantised values is enormous and its integral over a bin is not approximated by the density at the centre. The reported likelihood is then too high by an amount that depends on how sharply the model has fit the modes, and the error grows as training progresses. The symptom is a bpd that keeps improving while sample quality does not, which is the classic signature of a model that has learned the discretisation rather than the data.

The amount of dequantisation noise matters. Too little and the model still fits modes; too much and the training distribution is smoothed beyond the data’s structure. The standard choice is uniform noise on the bin width, which is the maximum entropy distribution consistent with knowing only the bin. Adding noise of a different width changes the implied data distribution and therefore the likelihood, so two models evaluated with different dequantisation noise are not comparable even if both report bpd.

Where the diffusion bound fits in

A diffusion model’s training objective is a bound on the log likelihood, obtained by applying Jensen’s inequality to the reverse process. The bound has a term for each timestep, and its value depends on the schedule, the parameterisation, and the terminal distribution. Converting that bound to bpd requires the same discretisation correction, and it requires care about which quantity is being bounded.

The bound is on the log likelihood of the continuous data distribution, not the discrete one. Applying the bin-width correction gives a bound on the discrete likelihood, and it is a valid lower bound because the correction is a constant added to both sides. What is not valid is comparing a diffusion bound computed with a continuous-time formulation against a discrete-time autoregressive model’s exact likelihood without checking that both have been corrected and that both use the same bin width.

The terminal term is where diffusion bounds lose tightness. The reverse process starts from a standard normal, but the true terminal distribution of the forward process is the data convolved with the largest noise scale, which is not standard normal. The KL between the two appears in the bound and is not small unless the schedule pushes the terminal state close to pure noise. This is one reason diffusion likelihoods are competitive but not dominant against autoregressive models on the same data: the bound pays for a mismatch that the autoregressive model does not have.

import numpy as np

def bpd_from_log_prob(log_prob_nats, num_values):
    # log_prob_nats is the log probability of the discrete sample
    return -log_prob_nats / (np.log(2.0) * num_values)

def bpd_from_continuous(log_density_nats, num_values, levels=256):
    # log_density_nats is the log of a continuous density at the sample
    bin_width = 1.0 / (levels - 1)
    log_prob = log_density_nats + num_values * np.log(bin_width)
    return bpd_from_log_prob(log_prob, num_values)

def uniform_baseline(levels=256):
    return np.log2(levels)      # exactly 8.0 for 8-bit data

The third function is the reference point that makes any bpd interpretable. A model at 8.0 has learned nothing; a model at 3.0 has removed five bits per value, which for a natural image is plausible for a good model and implausible for a weak one. Reporting a number below the entropy of the data would indicate an error in the accounting rather than an exceptional model.

What the number does not measure

Bits per dimension measures the average code length the model assigns to the data it was evaluated on. It does not measure sample quality, and the two can move in opposite directions. A model can achieve a low bpd by assigning high probability to the training distribution’s mode structure while producing samples that are blurry, because the likelihood rewards covering all the data and the sampler’s fidelity depends on how the model allocates mass within each mode.

The number is also sensitive to the evaluation set’s size. A model evaluated on a small held-out set can have a bpd that reflects memorisation of those examples, and the variance across evaluation subsets can be larger than the differences between models. Reporting a bpd without the evaluation set’s size and the dequantisation scheme is reporting a number that cannot be reproduced.

Finally, bpd is not comparable across data types. The same value on an 8-bit image and on a quantised audio waveform means different things, because the number of levels and the correlation structure differ. Comparisons are meaningful within a dataset and a preprocessing pipeline, and the pipeline includes the resizing, the colour space, and the quantisation. Two models evaluated with different resizing filters are not being compared on the same data, even if both report bpd on the same nominal dataset.

The likelihood accounting interacts with the compression stage described in latent diffusion: a model that generates in a compressed space has a likelihood in that space, and converting it to a likelihood on images requires the autoencoder’s Jacobian, which is not computed in practice. The bound itself, and the terms it is built from, are developed in the ELBO derivation; the discretisation correction applies to that bound in exactly the same way.