Early Depth Tests and the Ordering Constraints They Impose

The output merger compares a fragment’s depth against the depth buffer and either writes the fragment or discards it. In the conceptual pipeline the comparison happens after the fragment shader, because the shader may have written to gl_FragDepth or discarded the fragment, and both change what the comparison should see. Running the comparison before the shader is therefore an optimization that requires proof the shader cannot invalidate it. Getting that proof, and losing it, is the subject of a large fraction of the performance advice attached to depth testing.

What early rejection saves

A fragment that fails the depth test costs, in the ordered pipeline, the full price of its shader: texture fetches, arithmetic, and the register pressure that came with it. Rejecting it before the shader replaces that cost with a depth read and a compare. For a scene with heavy overdraw, the number of fragments that would be shaded and then rejected can be several times the number of visible pixels, so the saving is proportional to the depth complexity of the scene rather than to its resolution.

The saving is only available when the fragment’s depth is known before shading. Depth after the perspective divide is an interpolant, and interpolation is done by the rasterizer as described in the varying path, so it is known at fragment generation. A shader that leaves depth alone preserves that value, and the comparison can be moved forward. A shader that writes depth produces a value known only after it runs.

There is a second, cheaper form of early rejection that does not need the shader contract at all: a hierarchical depth buffer. The depth buffer is maintained with a coarse summary per tile, usually the maximum depth stored in it, and a whole tile is rejected if the incoming primitive’s nearest depth exceeds that maximum everywhere. Greene, Kass and Miller’s hierarchical Z-buffer is the canonical description, and the modern incarnations under the name Hi-Z in hardware and in software occlusion culling are the same idea with the summary updated per tile. Tile-level rejection needs no per-fragment proof: it rejects only tiles where every pixel would fail, which is true regardless of what the shader does.

The contract that makes early depth legal

For a per-fragment early test, the pipeline has to know that the depth value at the time of the test equals the value at the time of the final comparison. That is guaranteed when the fragment shader declares no depth output, does not discard, and does not write to storage that aliases the depth buffer. Vulkan expresses this with an early_fragment_tests execution mode that the shader can declare, which forces the depth and stencil tests to run before the shader body; OpenGL has an equivalent through the fragment shader’s early test control, and both APIs specify the same constraint: the shader must not write depth.

Discard is the harder case. A shader that discards fragments conditionally means the depth comparison’s outcome depends on the shader’s decision, so the test cannot be moved earlier. Implementations do not generally allow a shader that uses discard to declare early tests; the attempt is a compile error, not a silent degradation. This is the right design, because a silent fallback would make the optimization’s presence depend on the driver, and a program whose correctness depends on the ordering would fail unpredictably.

The frequent mistake is a shader that writes depth conditionally, or writes it to the same value it would have interpolated to. The latter is semantically a no-op but forces the slow path, and the fix is to remove the write rather than to hope the compiler sees through it. The consumer-level symptom is a pipeline that is unexpectedly slow only on hardware that respects the contract, which is the worst kind of portability difference: it is not an error, so nothing reports it.

#version 450
// Depth is not written and nothing is discarded, so the driver may run the
// depth test before this shader. Declaring early_fragment_tests requests it.
layout(early_fragment_tests) in;

layout(location = 0) in vec2 uv;
layout(location = 0) out vec4 color;
layout(binding = 0) uniform sampler2D albedo;

void main()
{
    color = texture(albedo, uv);
}

Depth prepass and the cost of proving order

The standard technique for scenes where shaders are expensive is a depth prepass: render the geometry with a trivial shader that writes depth only, then render again with the full shader and an equal-depth test. The prepass fills the depth buffer with the nearest surface, so the main pass rejects every hidden fragment at the early test, and because the prepass shader is trivial it is cheap per fragment and cheap to shade.

The prepass costs a full geometry pass and the depth bandwidth for every fragment it writes, whether or not it survives. On geometry that is mostly visible the prepass is pure overhead; on geometry with high depth complexity it pays for the main pass many times over. The break-even point is not a fixed ratio because it depends on the shader’s cost relative to the fixed cost of fragment generation, and that ratio changes with resolution and with shader complexity. This is why the technique is applied per material class rather than globally.

The prepass interacts badly with reversed-Z unless the comparison function matches it, and interacts well with the equal-depth test because a fragment that passes the prepass’s exact depth will pass an equal test exactly, with no epsilon needed. That equality is only reliable when both passes interpolate depth identically, which they do only if the rasterizer evaluates the interpolant at the same position and the depth values are stored without lossy compression. Hardware depth compression is lossy in some formats, and an equal test against a compressed-then-decompressed value can fail for fragments that should pass; the usual mitigation is to use a less-or-equal test rather than equal.

Multisampling and the per-sample complication

Under multisampling the depth test is per sample, and a fragment can pass at some samples and fail at others. The early test then has three outcomes rather than two: reject entirely, accept entirely, or partially accept. Only the first is a clean saving; the third still needs the shader to run because some samples survive. Hardware computes a coverage mask from the per-sample comparison and passes it along, and the shader’s output is written to exactly those samples.

The consequence is that the saving from early rejection is smaller under multisampling, because partial acceptance is common along every silhouette and every shared edge. Conservative rejection is still worthwhile for fully hidden fragments, and the tile-level summary in a hierarchical depth buffer is unaffected because it is per tile rather than per sample. The cleanest way to think about it is that the early test under MSAA rejects at the granularity of the coverage mask rather than of the fragment.

The interaction with analytic coverage is even more awkward: a fragment with fractional coverage has a single depth value but partial coverage, and the depth test must decide against which samples it applies. Storing fractional coverage in the depth buffer is not possible in a fixed-point per-pixel format, so the composite structure keeps coverage separate from depth and the test operates on the sample bits. That is one of the reasons hardware A-buffer variants keep a sample mask rather than a true area.

Ordering, blending, and the boundary of the optimization

Early depth is incompatible with any operation whose result depends on the order in which fragments are processed. Blending is the main one: a fragment that fails the depth test must not have blended its colour, because the blend would have modified the target before the rejection. Since early testing decides rejection before the shader runs, no blend can occur, which is consistent. The incompatibility runs the other way: a fragment that passes must still blend in a defined order, and early testing does not change that order, so the two features coexist.

The incompatibility appears when a shader writes to a resource that another draw reads, or when the depth test’s outcome depends on a value that the same draw writes. Depth writes within a draw are the classic case: a triangle’s early test sees the depth buffer as it was at the start of the draw if the implementation has deferred depth writes, or as it is after previous triangles of the same draw. The specification constrains this: depth writes are ordered with respect to subsequent depth tests, so a later triangle sees the effect of an earlier one. Implementations satisfy that by tracking that they must be ordered, and the tiled design is what makes the tracking tractable, because all triangles affecting a tile are processed together and the order is local.

Measuring depth complexity

The quantity that decides whether any of this matters is the depth complexity, the average number of fragments generated per pixel. A value near one means almost nothing is hidden and early rejection saves nothing. A value of five or ten, common in scenes with layered geometry, means the main pass could be doing a fifth of the shading. The number is a property of the scene and the camera, not of the rasterizer, and it is not directly observable through the graphics API in a portable way; occlusion queries give counts for chosen regions, which is usually enough to find the offending passes rather than to instrument the whole frame.

The complementary metric is the number of fragments rejected at the tile level, which indicates whether the hierarchical structure is doing its job. A scene with a badly constructed depth range, for instance one where the near plane is set far too close, has a depth buffer that is mostly empty at the levels the tiles summarize, and the tile maxima stay high, so tile rejection finds nothing. The fix there is the projection rather than the rasterizer, which is exactly the problem that reversed-Z addresses by making the far field distinguishable at all.