Winding Order, Backface Culling, and the Two-Sided Lighting Problem

Every triangle submitted to the pipeline carries an implicit orientation: the order in which its three vertices appear. The orientation is not geometric information in the usual sense, because the three points are the same set regardless of order; it is a convention that says which side of the triangle’s plane is the outside. The GPU uses that convention in two places, and the two uses have different failure modes: the rasterizer’s signed-area test for culling, and the interpolation stage’s choice of sign for face normals.

The signed area test

For a triangle with screen-space vertices a, b, c, the doubled signed area is the determinant

A = (b.x - a.x)(c.y - a.y) - (b.y - a.y)(c.x - a.x)

The magnitude is twice the triangle’s area. The sign is the orientation: positive for one winding direction and negative for the other. In a coordinate system whose y axis points down the screen, which is the convention for render targets, a vertex loop that appears clockwise on screen has one sign and a counter-clockwise loop the other. Positive area in this convention corresponds to clockwise ordering, which is the origin of the “clockwise is front-facing” default in several APIs’ documentation and the reason the same mesh can appear inside-out when ported between APIs whose y-axis conventions differ.

The test is cheap because the rasterizer computes this determinant anyway. The edge functions used for coverage testing, E(p) = (b.x - a.x)(p.y - a.y) - (b.y - a.y)(p.x - a.x), are that same determinant with p substituted for c. So the area is the third edge function’s value at the third vertex. There is a further identity: a triangle’s signed area equals the value of the first edge function evaluated at the third vertex, equals the second at the first, equals the third at the second. An implementation that computes the coverage test in normalised form — flipping all three edge functions when the area is negative — has already consumed the sign and can report it as a side effect at no extra cost.

/* Front-face determination from the unnormalised edge functions.
   Screen space with y increasing downward, as in a render target. */
#include <stdbool.h>
#include <stdint.h>

typedef struct { int32_t x, y; } v2;

static int64_t cross(const v2 a, const v2 b, const v2 c)
{
    return (int64_t)(b.x - a.x) * (c.y - a.y)
         - (int64_t)(b.y - a.y) * (c.x - a.x);
}

/* `ccw_front` is the pipeline's front-face state: true if the API has
   been told that counter-clockwise loop order marks a front face. */
bool is_front(v2 a, v2 b, v2 c, bool ccw_front)
{
    int64_t area2 = cross(a, b, c);
    if (area2 == 0)
        return false;             /* zero-area triangle: neither side */
    return (area2 > 0) != ccw_front;
}

The zero case deserves attention. A triangle whose vertices are collinear in screen space has zero area and is neither front nor back; it covers no sample point under the fill rules of rasterization rules and the top-left fill convention, so the answer does not affect output, but it does affect internal counters and any query that counts primitives generated. Reporting such a triangle as back-facing and culling it discards it earlier, which is usually preferable but is not what the specification requires — the specification permits discarding it at rasterization.

Where the test runs

Culling is a rasterizer operation performed per triangle, after clipping and after the perspective divide, because the sign of the projected area is what matters. It cannot be done in the vertex shader, since a vertex shader sees one vertex at a time and has no knowledge of the triangle it belongs to. The orientation it tests is the one the divide produces: for a triangle that straddles the eye plane, w changes sign across the triangle, so the loop order in clip space does not correspond to the loop order on screen.

The placement has a cost consequence: culling happens after the vertex shader has run for all three vertices and after transform feedback and geometry shading, so a culled triangle has already paid the full vertex-stage cost. For a closed mesh with half its triangles facing away, that is half the vertex work paid for nothing. Frustum culling, which rejects whole objects before their vertices are submitted, is the coarser and more valuable operation, and the two compose: a mesh behind the camera is rejected for the price of a bounding-sphere test, whereas a mesh with half its faces away still costs one vertex shader run per face. Homogeneous clipping and the guard band covers the related question of how much of the triangle pipeline can be skipped for geometry outside the view.

A geometry shader or a mesh shader that knows the triangle’s vertices can cull before the vertex work is fully paid, which is one of the few reasons to use those stages on hardware where they are supported. The savings are real but come with their own throughput limits, and the programmability is rarely worth it for simple culling on hardware whose fixed-function path is faster.

Culling also has a two-sided variant: the pipeline can be told to keep front faces, keep back faces, or keep both, and it can be told to reverse the face orientation (sometimes called front-face flipping) for the duration of a draw. The reversal exists for the case of geometry that is authored with the opposite convention, such as a mesh imported from a renderer whose y axis points up: flipping is cheaper than transforming the mesh on the host.

Normals and the two-sided surface

The face orientation has a second consequence, entirely separate from culling. A flat-shaded triangle’s normal is derived from the same signed area, and flipping the winding flips the normal. For a closed, consistently wound mesh with both faces kept, this is harmless because the visible side is front-facing and its normal points out. For a two-sided surface — a leaf, a sheet of paper, a flag, a shadow-casting plane — both sides are visible, and the back side’s normal points into the surface. Lambertian shading with a back-facing normal produces a dark triangle where the light comes from the visible side.

The shading language provides a built-in that reports which face the fragment belongs to, and implementations supply it from the rasterizer’s orientation test rather than from a geometry calculation. The usual fix in a two-sided shader is to negate the interpolated normal when the face is back-facing, which restores a normal that points toward the viewer. The negation is not sufficient by itself: it flips the normal for lighting but not the fog or the ambient occlusion term, which may have been authored with the assumption that normals point away from the viewer. Shaders that handle two-sided surfaces correctly negate the normal at the point where it enters the lighting calculation and leave the geometric normal untouched elsewhere.

The subtlety with interpolation is that if the vertex normals are used for smooth shading, the face orientation and the interpolated normal can disagree. A vertex normal is an authored value, not derived from the triangle, so a triangle wound backwards with correct vertex normals will shade correctly under mirrored lighting terms and incorrectly under the linear interpolation of perspective-correct interpolation, which is the reason rendering two-sided surfaces is usually done by duplicating the leaf geometry with reversed winding and reversed normals rather than by relying on the sign test.

Inconsistencies that survive to the screen

The most common winding bug is not a culling bug but a mesh bug: a model whose faces are inconsistently wound. Half the faces are culled when they should be visible and half are not culled when they should be, and the visible symptom is holes in the surface that move as the camera moves, because which half is wrong depends on the viewing direction. The bug is invisible in a wireframe view and invisible in a render with culling disabled, which is why it surfaces only when culling is first enabled.

The second common inconsistency is between the winding a mesh is authored with and the winding the pipeline expects. The default differs across APIs and across the coordinate conventions they assume, and a mesh that looks correct in one API can appear inside-out in another with no change to its data. The visible symptom is the same holes; the diagnosis is to check the winding convention of the asset pipeline rather than the culling setting.

A third case is a mesh whose faces are wound consistently but whose vertex normals contradict the winding. This does not affect culling at all, since culling uses the projected area, but it does affect lighting and it does affect any computation that uses the normal to reconstruct a surface — tangent space generation, normal mapping, and shadow-map depth comparisons. The contradiction is silent in the pipeline, which is why mesh validation tools check both the winding and the normal direction of each face.