Ray Differentials for Texture Lookups Along a Ray

A rasterizer gets texture filtering for free because the hardware computes derivatives from neighbouring fragments. A ray tracer has no neighbouring fragments. Each ray is launched independently from a pixel, and when it hits a surface the question of how much texture area that pixel covers has no local answer: the footprint depends on the camera’s projection, on the distance to the surface, on the curvature of the surface at the hit, and on every reflection or refraction the ray passed through to get there. Choosing a mip level without that information gives either aliasing or an arbitrary constant blur, and both are obvious on a reflective floor.

The differential as a first-order propagation

A ray differential is the derivative of the ray with respect to the screen coordinates it originated from. Rather than one ray, the renderer carries three: the primary ray and two auxiliary rays offset by one pixel horizontally and one pixel vertically. Each has an origin and a direction, and the differential is the difference between the auxiliary ray and the primary, expressed at the same parameter $t$.

At a hit point, the differentials of the hit position and of the texture coordinate follow by propagating through the intersection. For a ray $o + t d$ and a differential $(\Delta o, \Delta d)$, the offset at parameter $t$ is $\Delta o + t \Delta d$ to first order, and the surface intersection point’s differential is the component of that offset lying in the surface’s tangent plane:

$$\Delta p_\perp = \Delta o + t,\Delta d - \big((\Delta o + t,\Delta d) \cdot n\big), n$$

The texture coordinate differential follows from the surface parameterization: $\Delta u = \nabla u \cdot \Delta p_\perp$, with $\nabla u$ the gradient of the texture coordinate with respect to surface position. For a triangle mesh the gradient is constant per triangle and can be precomputed from the vertex positions and UVs; for a parametric surface it comes from the partial derivatives of the parameterization. The two differentials $\Delta u$ and $\Delta v$ are the same quantity the rasterizer derives from neighbouring fragments, obtained by a different route.

The similarity is exact enough that the level computation is unchanged: convert to texel units, take the maximum of the two components of each differential, and take the logarithm base two. The machinery described in deriving mip level from screen-space derivatives applies verbatim once the differentials exist. What differs is where they come from and what happens to them at a bounce.

Reflection, refraction, and the collapse

At a specular bounce the ray direction changes, and the differential must change with it. The reflection of direction $d$ about normal $n$ is $r = d - 2(d \cdot n)n$. Differentiating, and accounting for the fact that the normal itself varies across the surface, gives a term for the direction change plus a term for the normal’s variation:

$$\Delta r = \Delta d - 2\big((\Delta d \cdot n) + (d \cdot \Delta n)\big) n - 2(d \cdot n),\Delta n$$

The second and third terms are the important ones. The term with $\Delta n$ accounts for the surface’s curvature: on a curved mirror, adjacent pixels hit points with different normals, and the reflected rays diverge or converge accordingly. Ignoring it makes a reflection on a sphere use the footprint of the flat tangent plane, which underestimates divergence near the silhouette and overestimates it at the centre.

The collapse case is a mirror that focuses. A concave mirror maps a range of incoming directions onto a smaller range of outgoing ones; at the focal point the differential goes to zero, and just past it the differential’s sign flips. A zero differential means the footprint is a point, the computed level is negative infinity before clamping, and the sampler returns level 0: sharp, aliased, and wrong, because the pixel actually covers a large region of the reflected environment near a caustic. Sign flips are worse: the differential passes through zero and the footprint’s orientation reverses, which no scalar level can represent.

Refraction adds a second mechanism. Snell’s law scales the transmitted direction by the ratio of indices of refraction, and the derivative of that scaling is a factor that is not one. Entering a denser medium compresses the angular spread; leaving it expands the spread. A ray that enters glass and exits on the far side accumulates two such factors and, for a curved surface, two curvature terms. The total differential can be an order of magnitude larger or smaller than the entry differential, and the level computed at the exit surface reflects the accumulated compression rather than the local geometry.

Clamping and the alternatives

Every production ray tracer clamps the differential. The differential can grow without bound in a long chain of bounces, can shrink to zero, and can become numerically meaningless when the hit point is nearly parallel to the surface. A minimum magnitude and a maximum magnitude, applied to the texel-space differential before the logarithm, bound the level to a usable range. The clamp is not a hack around a bug; it is a statement that the first-order approximation is only valid over a limited range, and beyond that range the true footprint is not representable.

/* Clamp a texel-space differential pair, then derive the mip level.
   uv_dx, uv_dy are per-pixel changes in normalized coordinates. */
float lod_from_differentials(float uv_dx[2], float uv_dy[2],
                              float texel_w, float texel_h)
{
    float tx[2] = { uv_dx[0] * texel_w, uv_dx[1] * texel_h };
    float ty[2] = { uv_dy[0] * texel_w, uv_dy[1] * texel_h };

    float lx = fmaxf(fabsf(tx[0]), fabsf(tx[1]));
    float ly = fmaxf(fabsf(ty[0]), fabsf(ty[1]));

    /* Clamp before the log so that degenerate footprints stay finite. */
    const float kMin = 1.0f / 16.0f;   /* no sharper than 1/16 texel per pixel */
    const float kMax = 1024.0f;        /* no blurrier than 1024 texels per pixel */
    float rho = fminf(fmaxf(fmaxf(lx, ly), kMin), kMax);

    return log2f(rho);
}

The clamps have visible consequences at both ends. A minimum below one texel per pixel allows magnification, which is correct for a surface close to the camera but produces aliasing when the differential is artificially small because of a collapse. A maximum that is too generous blurs reflections to a flat colour near silhouettes and grazing angles; too small and the same regions alias. The values are tuned per scene rather than derived, because the correct value depends on how much geometric detail the environment map contains.

There are alternatives to first-order differentials. The simplest is to ignore them entirely and pick a level from the ray’s distance to the hit point, which is correct for a flat surface viewed head-on and wrong everywhere else, but is cheap and stable. The next is to take the footprint from a screen-space estimate after the fact, which is what a denoiser or a screen-space reflection pass can do because it has neighbouring pixels. Neither handles the curved-mirror case; both handle the common case of a floor with a distant reflection well enough that the artifact is invisible.

Supersampling is the honest alternative: cast several rays per pixel and average. That measures the footprint’s effect without modelling it, at a cost proportional to the sample count, and it handles every nonlinear case that differentials miss. The reason differentials remain useful is that they cost two extra rays per bounce rather than several samples per pixel, and they let a sampler choose a level that is roughly right, which reduces the variance a supersampled renderer has to average away. A ray tracer that uses differentials plus a modest sample count converges faster than one that uses no differentials and more samples, on scenes with textured surfaces at distance.

Where the approach breaks structurally

Three failure modes are inherent to the method rather than to its implementation.

The first is that the differential is a linearization, and a linearization is wrong across a discontinuity. A pixel whose footprint spans a silhouette covers two surfaces with unrelated texture coordinates; the differential computed at the hit on one side says nothing about the other. The correct footprint is the union of two footprints, and the level should be the coarser of the two. Detecting the case requires knowing that the pixel’s cone intersected more than one surface, which the single-ray formulation cannot express. Practical renderers accept the error and rely on the fact that silhouettes are usually handled by geometric antialiasing rather than texture filtering.

The second is that the differential propagates through geometry but not through the environment. A ray that misses all geometry and samples an environment map has a differential in direction, not in position, and the map’s parameterization converts direction to coordinates with its own Jacobian. Near the poles of a latitude-longitude map, the horizontal coordinate changes rapidly with a small change in direction, so the differential is large and the level is coarse. That is correct for the map’s own sampling but wrong for the solid angle the pixel covers, and the poles therefore look blurred even when the environment detail is uniformly distributed. Cube maps avoid the pole problem by construction, which is one reason they are preferred for this use.

The third is storage. Carrying three rays per bounce triples the memory traffic of the ray queue, and the differential must be transformed at every hit, every bounce, and every medium transition. In a wavefront path tracer, the differential is another per-ray attribute that must be packed, sorted, and unpacked alongside the ray. The cost is small relative to shading but not small relative to traversal, and implementations often carry differentials only for the first bounce and fall back to distance-based levels for deeper paths, where the artifacts are hidden by accumulated blur.

What differentials buy beyond mip selection

The differential is not only useful for choosing a level. It gives the sampler the shape of the footprint, which is what anisotropic filtering needs: the two differential vectors are the axes of the pixel’s footprint in texture space, so their lengths give the anisotropy ratio and their direction gives the major axis. A ray tracer that carries differentials can therefore apply the footprint-shaped kernel described in anisotropic footprint filtering without any screen-space derivatives, which is exactly the information a rasterizer gets for free.

The same differentials drive texture coordinate derivatives for procedural shading, where they are needed to antialias a procedural pattern that has no mip chain. A noise function evaluated with the wrong footprint aliases; evaluating it with a band-limited approximation based on the differential’s magnitude is the only way to keep it stable. The differential is the general form of the rasterizer’s derivative, and it is available for every quantity the ray carries, not only for texture coordinates.

The last thing to note is that the level computed from a differential is a scalar and the footprint is not. Collapsing an elongated footprint to one level is the same compromise discussed in the anisotropic case, with the same consequence: the level matches the long axis and the short axis is over-blurred. Ray tracers have the additional option of clamping the ratio more aggressively than a rasterizer would, because the number of taps in a reflected footprint is multiplied by the number of bounces and because a reflection that is slightly too sharp is less objectionable than one that shimmers.