Filtering Normal Maps Without Flattening the Detail

A normal map is not an image. Its entries are coordinates of direction vectors packed into colour channels, and every operation that treats them as colours is wrong in a way that is easy to miss, because wrong normals still produce plausible lighting. Averaging two normals whose y components have opposite signs gives a normal pointing sideways, and the same reduction that produces a correct average colour in mip chain construction produces a vector that no longer describes a surface; averaging a tight cluster of normals pointing in nearly opposite directions gives one pointing nowhere in particular, of length far below one. The visible result is a surface that flattens as it recedes and then breaks into patches, and the break is not where the detail ends but where the sampling rate crosses the level at which the averaged vector loses its direction.

What a mip level of normals actually means

A mip level of an albedo texture represents the average colour of a region. A mip level of a normal map should represent the distribution of surface orientations in that region, and a single vector cannot represent a distribution. This is the central difficulty: the level-of-detail reduction destroys exactly the information that matters for shading, which is not the mean direction but the spread around it.

Consider a region containing a fine corrugation, like a brushed metal surface. The normals alternate between two directions symmetrically about the geometric normal. The arithmetic mean of the two is the geometric normal, with length equal to the cosine of the tilt angle. Renormalizing that mean gives exactly the geometric normal, which is the correct mean orientation. But shading a microfacet surface with a single normal at roughness $\alpha$ gives a specular lobe of width $\alpha$, whereas the true surface has a lobe that is the convolution of the original lobe with the distribution of the corrugation. The mean direction has been preserved and the width has been lost, so the surface becomes too glossy at distance. The highlight narrows and sharpens, which is the opposite of the usual filtering artifact and is why normal-map mips look wrong in a way that is hard to attribute.

The standard answer is the normal distribution function’s variance. Under the Beckmann or GGX microfacet model, roughness is the width of a distribution over half-vectors, and two independent contributions to that width combine in quadrature: the material roughness and the geometric variation of the normals. If a region’s normals have a variance $\sigma^2$ about their mean, the effective roughness for that region satisfies

$$\alpha_{\text{eff}}^2 = \alpha^2 + \sigma^2$$

approximately, for small angles and for the Trowbridge-Reitz form. A filtered set of levels therefore needs two channels from the normal map, the mean direction and the variance, and one channel from the roughness map, combined per level. Storing only the mean and renormalizing gives the first term right and the second term zero.

Building the two statistics

The mean direction is the normalized average of the unit normals in the region, which is the vector sum. Weighting by solid angle is more correct than weighting by texel count when the region spans a wide angular range, but for a tangent-space map whose normals stay within a hemisphere, the plain sum is a reasonable approximation and is what any reduction pass computes.

The variance is the mean of the squared angular deviation from the mean direction. For unit vectors, the quantity $|\bar{n}|$ of the normalized mean gives it directly: for a cluster of unit vectors, the length of the mean vector is $|\bar{n}| = \sqrt{1 - \sigma^2}$ for small isotropic spreads, so $\sigma^2 \approx 1 - |\bar{n}|^2$. That identity is convenient because it needs no extra channel: the reduction computes the vector sum, and the length of the sum before normalization carries the variance.

/* One texel of a filtered normal map: mean direction plus variance. */
typedef struct { float x, y, z, var; } normal_texel;

/* Reduce a 2x2 block of unit tangent-space normals into one texel.
   The four inputs arrive already decoded and normalized. */
normal_texel reduce_normals(const float n[4][3])
{
    float sum[3] = {0, 0, 0};
    for (int i = 0; i < 4; ++i)
        for (int k = 0; k < 3; ++k)
            sum[k] += n[i][k];

    float mean[3] = {sum[0] / 4, sum[1] / 4, sum[2] / 4};
    float var = 1.0f - (mean[0] * mean[0] + mean[1] * mean[1] + mean[2] * mean[2]);

    float len = sqrtf(sum[0] * sum[0] + sum[1] * sum[1] + sum[2] * sum[2]);
    normal_texel out;
    out.x = sum[0] / len;          /* renormalized mean direction */
    out.y = sum[1] / len;
    out.z = sum[2] / len;
    out.var = var > 0.0f ? var : 0.0f;
    return out;
}

The reduction must be done on decoded vectors, not on packed bytes, and it must accumulate in floating point across the whole pass. A 16-bit-per-channel storage format gives enough precision to decode and renormalize; an 8-bit format quantizes each normal to about 1/128 of the component range, and summing hundreds of quantized values produces a mean whose own quantization error is comparable to the variance being measured for smooth regions.

The chain itself must be built from the level below, which means each level’s variance is a recursive combination of the previous level’s variance plus that level’s local variance. Because variances add under independent combination, this recursion is well behaved and does not need access to level 0. Each step doubles the covered area, so the variance grows monotonically up the chain, which is the desired behaviour: more area, more variation, wider effective lobe.

The filtering trade in the sampler

Whether the runtime uses the variance channel depends on what it can do with it. Sampling a normal map with bilinear filtering at a level chosen for minification gives the interpolated vector, whose length encodes the local variance implicitly. A shader that normalizes the interpolated vector discards that information. A shader that uses the length as a roughness contribution keeps it:

// Sample at an explicit level and derive roughness from the vector's length.
vec3  n = textureLod(normalTex, uv, lod).xyz * 2.0 - 1.0;
float len = length(n);
vec3  nrm = n / max(len, 1e-4);
float var = clamp(1.0 - len, 0.0, 1.0);        // local variance, small-angle form
float rough = sqrt(baseRough * baseRough + var);

This is the cheapest possible version and it uses only what the interpolation already provides. Its accuracy degrades when the interpolated normal is far from a mean of four vectors, which happens at level boundaries and in regions where the normal changes sharply, because the eight samples that go into a trilinear fetch are not a uniform sample of the region. A cleaner build stores the variance explicitly in a channel and combines it at the point of use, which costs one extra channel per texel and one extra fetch.

The two approaches differ in how they fail. The implicit version needs no extra storage and degrades smoothly as the level increases, but it systematically underestimates variance when the stored normals are nearly opposed, because their sum is short in a direction that is not the mean of the distribution. The explicit version is exact per level, but the variance is only meaningful if the level was built with the same angular measure the shader uses, and mixing implementations is a common source of levels that are too rough or too smooth.

Either way, the renormalization must happen after the level reduction, not before. A build that normalizes every texel at authoring time and then generates mips with a plain average produces mips whose vectors are shorter than one, which then need renormalizing at sample time; that is equivalent to the implicit method and is why simply adding a normalize after the mip build already improves the appearance substantially without any variance channel.

Detail normals and the two-scale problem

Most surfaces use two normal maps: a base map at low frequency describing large forms, and a detail map at high frequency describing tooling, grain, or fabric weave. The detail map is tiled at a much higher rate than the base, so it is minified sooner. Combining the two before filtering is not the same as filtering them separately and combining after, because the reduction is nonlinear in the normalized sum.

The correct order is to filter each map to its own level, then combine the two mean directions and add their variances. Combining the variances requires the two distributions to be independent, which is usually the intent of a detail map: it adds grain without changing the large form. When the detail is strongly correlated with the base, such as scratches aligned to a seam, the variance sum overestimates and the surface becomes too rough in those areas.

A further subtlety applies to the scale of the detail. A detail normal map tiles at, say, 8 repeats across the surface, so its level of detail must add $\log_2 8 = 3$ to the level computed from the base UVs. Getting that offset wrong is a factor-of-eight error in the sampling rate, which is severe: the detail either aliases badly or vanishes entirely. The correction is a per-map constant added to the derived level, and it must also be applied to the anisotropy ratio, because tiling changes the footprint size without changing its shape.

What breaks and what to leave alone

Reconstructing the third component from two channels is incompatible with averaging. A two-channel map stores $x$ and $y$ and rebuilds $z = \sqrt{1 - x^2 - y^2}$, which is exact per texel but not linear in the stored values: the average of two valid $(x,y)$ pairs does not correspond to the average of their $z$ components. Filtering the two channels independently gives a vector whose reconstructed $z$ is larger than the mean $z$ when the tilts point in opposite directions, so the filtered normal is flatter than the true mean. The error is bounded and small for gentle detail and grows for strong detail, and it is the reason tangent-space encoding of two channels is standard for uncompressed storage and less common for maps that need heavy filtering.

Object-space normal maps have the opposite problem. Their vectors are spread across a hemisphere or more, so any region spanning a silhouette or a hard edge has a mean length near zero and no meaningful direction. Filtering them per level is close to meaningless along geometry seams; the practical approach is to keep object-space maps at high resolution and accept aliasing at distance, or to resample them into tangent space where the local frame makes the distribution narrow.

The safe rule for what not to filter: normals across UV seams, normals across material boundaries where the underlying geometry actually creases, and normals in regions where the map encodes a different surface than its neighbours. These are discontinuities in the function, not detail within it, and any averaged level is a lie about the geometry. Splitting the mesh at the crease so that no triangle and no mip block spans it is the only correct remedy, and it is the same remedy the wrap seam problem needs in texel grids and wrap addressing.

Finally, specular antialiasing from a normal map is not solved by normal-map filtering alone. Even a perfectly filtered distribution produces highlights that vary within a pixel when the distribution is wide, and the correct treatment evaluates the specular integral against the filtered distribution rather than against a single normal. That is a per-pixel integral, and the approximation used in practice is to widen the roughness and to attenuate the specular term by the fraction of the distribution that faces away, which is where the variance channel described above becomes the natural input.