Early Depth Testing and the Order of Per-Fragment Operations

The per-fragment stage of a graphics pipeline is described by the specifications as a fixed sequence: scissor test, stencil test, depth test, blending, and dithering, with the fragment shader’s output feeding into the sequence at a defined point. That sequence is the contract. The order in which an implementation actually performs its work is a different matter, because the depth test in the middle of the sequence can save an enormous amount of work when performed much earlier — and moving it is only legal when the shader’s behaviour makes the move unobservable.

The specified order and why it is that order

The sequence in the OpenGL specification, for the fragment that survives rasterization, is roughly: pixel ownership and scissor test, fragment shader (or shader followed by early tests if the shader opts in), stencil test, depth buffer test, occlusion query accounting, blending, dithering, and then the framebuffer write. The stencil and depth tests reject fragments before blending, so a rejected fragment costs no blend and no write.

The ordering exists because the tests are cheap and the write is not. A depth test against a cached depth value costs a comparison; a blend costs a read of the destination, arithmetic, and a write. The specification puts the cheap rejection first so that the expensive part applies only to fragments that change the framebuffer. This is the original design constraint, and it predates programmable shaders: in a fixed-function pipeline, the fragment colour was computed by machinery no more expensive than the tests, so putting the tests first was unambiguously good.

Programmable shaders broke that assumption. A fragment shader can be a hundred instructions, sample several textures, and take a variable number of cycles depending on divergence. When the shader became the most expensive item in the stage, having it run for fragments that the depth test would have rejected became the dominant cost — the cost of overdraw. On a scene with a depth complexity of four, three quarters of the shading work is wasted if every fragment is shaded before it is tested.

Hoisting the test

The fix is to run the depth test — and the stencil test, and the scissor test, and the early-fragment-tests depth write — before the shader, when the shader’s source makes that safe. The conditions under which it is safe are specific and conservative.

The shader must not modify its own depth output. A shader that writes a depth value other than the interpolated one is computing a new depth, and the test must use that value, so the test cannot precede the computation. In current APIs the shader declares whether it writes depth — the gl_FragDepth output in OpenGL, the corresponding output in SPIR-V, or the absence of a depth output in Vulkan’s model — and the compiler can see the declaration. A shader that writes depth in one branch and not another forces the conservative answer.

The shader must not discard. discard removes the fragment, so a test performed before the shader that wrote a depth value would have committed a write that the shader later revokes. Implementations can sometimes analyse the shader and hoist when the discard is provably not taken, but a shader that discards conditionally on interpolated data — the usual case — cannot be hoisted.

The shader must not perform side effects that the specification requires to happen exactly once per covered fragment, in particular atomic operations and image or buffer writes. A hoisted test that rejects a fragment would skip the shader, and therefore the side effect, which is not the specified behaviour. The specification is blunt about this: the ordering of side effects with respect to the test is observable, and implementations must not change it.

#version 450
// A shader that permits early depth testing: no depth output, no discard,
// no side effects before the final colour is computed.
layout(location = 0) in vec3 normal;
layout(location = 1) in vec2 uv;
layout(location = 0) out vec4 frag;

void main()
{
    // The depth test may run before this executes; fragments that fail
    // never reach this code, and the framebuffer sees no change.
    vec3 n = normalize(normal);
    float lambert = max(dot(n, normalize(vec3(0.3, 0.8, 0.5))), 0.0);
    vec4 texel = texture(uv);
    frag = vec4(texel.rgb * lambert, 1.0);
}

APIs express the requirement as an explicit declaration rather than leaving it to inference. OpenGL has a layout qualifier, early_fragment_tests, that requires the tests to execute before shading, and SPIR-V — the form Vulkan consumes — has an execution mode with the same meaning. There are also execution modes that state how a shader’s written depth relates to the interpolated value, greater, less or unchanged, which is what lets an implementation keep the test early for a shader that does write depth. The declaration binds the implementation; without it, whether the tests run early is an optimisation applied when the implementation can show the shader permits it.

What early testing actually saves

The saving is proportional to the overdraw, and overdraw is a property of the scene rather than of the hardware. A scene rendered front-to-back with a depth buffer per triangle amortises the early test across many rejections; a scene rendered back-to-front gets nothing, because every fragment passes the test until the nearest one is drawn, which is the last.

Two implementation realities modulate the saving. The depth test reads and possibly writes the depth buffer, and the hardware caches depth values in a hierarchical structure — a compressed representation of depth per tile, sometimes called a depth pyramid or Hi-Z buffer — that lets it reject whole quads or whole tiles without per-fragment work. This hierarchical test is a coarse conservative test: it answers “is every fragment in this tile behind the stored maximum?” If so, the tile is rejected wholesale. If not, the fragments fall back to per-fragment testing, and the pyramid is updated on writes.

The consequence is that early depth testing can reject fragments at three granularities: per tile, per quad, and per fragment. The per-tile rejection is what makes a scene with heavy overdraw cheap, because it operates before any shader work and before any fragment-level bookkeeping. It also means that a depth buffer that is being written constantly in a scrambled order defeats the hierarchy: each write invalidates the tile’s cached maximum, forcing recomputation.

What it does not save

Early depth testing does not avoid rasterization. The fragment still had to be generated, the quad still had to be filled, and the interpolated attributes still had to be computed to feed the test. On a scene with small triangles, this setup work dominates and the early test’s saving is small.

It does not help with blending, because blending requires the shader’s output and cannot be reordered before it. A pipeline with blending enabled and depth testing on gets the early depth rejection for fragments behind existing geometry, but every fragment that passes pays the full blend. Alpha-tested foliage, which usually also discards, is the worst case: it cannot hoist the depth test because of the discard, so it shades everything and then discards most of it. The standard workaround is a depth pre-pass, which renders the opaque geometry only, with early tests, to establish depth, then renders the alpha-tested geometry against it — at which point the geometry behind the leaves is rejected at tile granularity. That doubles the vertex and rasterization work for the opaque pass and saves the shading for the occluded part of the alpha pass.

It does not help with fragment shading that depends on derivatives, because a hoisted depth test can reject individual lanes but the shader’s quad execution still needs covered lanes to have neighbours. Helper invocations are generated for uncovered positions regardless of the depth test’s verdict on the covered lane, because the test’s verdict is a per-fragment property and the quad’s structure is fixed before the test runs.

Ordering guarantees that remain

Even with early testing, the visible ordering constraints of the pipeline survive. Fragments of a single primitive are ordered among themselves, as specified by the coverage rules, so blending remains deterministic for overlapping geometry within a triangle. Fragments of different primitives retain the order in which the primitives were submitted, up to the point where the depth test’s rejection makes the order irrelevant. Writes to the framebuffer are visible to subsequent draws without an explicit flush, because the pipeline guarantees the ordering at the API level; the implementation may buffer internally, but it must satisfy the ordering at the boundary of each draw.

What does not survive is any assumption that a shader’s side effects happen for every fragment the rasterizer generated. A shader that counts invocations with an atomic increment and compares the count to the number of generated fragments will find the count lower when early testing rejects fragments, which is correct behaviour and a common source of confusion when instrumenting a renderer. The specification’s phrasing is that the shader executes for fragments that pass the tests, not for fragments that were generated.