Vertex Attribute Quantization and the Range It Depends On

A vertex position is a 32-bit float per component, twelve bytes for a VEC3. A unit normal is another twelve, a tangent sixteen, and two UV sets sixteen more. That is 56 bytes per vertex before any colour or skinning data, and for a mesh of a million vertices the attributes alone exceed 50 MB. Most of those bits are doing nothing: a float has 24 bits of significand, but a position measured in metres across a model ten metres across needs no more than about 16 bits to stay under a millimetre of error. Quantization is the process of spending exactly the bits the geometry needs and no more, and its central difficulty is that “the bits the geometry needs” is a property of the whole mesh, not of a vertex.

The scale-and-bias model

The scheme that dominates is uniform affine quantization: pick a per-mesh (or per-primitive) minimum and maximum for each component, map that range onto the integer range of the storage type, and record the mapping so the decoder can invert it.

For an unsigned $n$-bit type the encoding is

$$q = \operatorname{round}!\left(\frac{v - \text{min}}{\text{max} - \text{min}} \cdot (2^n - 1)\right)$$

and the decoder recovers $v$ as

$$v’ = \text{min} + q \cdot \frac{\text{max} - \text{min}}{2^n - 1}$$

The quantization step, meaning the distance between adjacent representable values, is therefore $(\text{max} - \text{min}) / (2^n - 1)$, and the maximum error is half of that, since rounding puts a value at most half a step from its nearest grid point.

That formula contains the whole trade. For 16-bit storage the divisor is 65 535; for 8-bit it is 255. A model spanning 10 metres quantized to 16 bits has a step of about 0.15 mm, and to 8 bits a step of about 39 mm. The 8-bit case is clearly inadequate for positions at that scale, but perfectly adequate for a model spanning a centimetre — which is the first important property: quantization error is relative to the mesh’s bounding box, not to the storage type alone. A small object gets fine precision from coarse storage, and a large one needs fine storage for the same absolute error.

This is also the first failure mode. The bounds must be computed over the actual stored values, and a pipeline that inherits them from a previous stage — a modelling package’s scene bounds, or the union of all meshes in a file — quantizes every mesh against the largest extent in the set. A pebble in a scene spanning a kilometre gets a step of 15 mm at 16 bits, and its silhouette becomes visibly faceted. Per-primitive bounds are the correct granularity; per-file bounds are the cheap approximation and the one that goes wrong.

Signed types, and why the negative range is asymmetric

Two’s-complement signed storage has an awkward property: an $n$-bit signed integer represents $-2^{n-1}$ through $2^{n-1}-1$, which is 65 536 values for $n = 16$ but with an uneven split around zero. Normalized signed integers in graphics APIs conventionally map the integer range onto $[-1, 1]$ by dividing by $2^{n-1} - 1$, the largest positive value, and clamping the most negative result. That is the same convention glTF uses for normalized accessors: the mapping is exact at both endpoints of the positive side and slightly over-ranged on the negative side.

The consequence is a small asymmetry in the error: the step near $-1$ is very slightly larger than the step near $+1$, and a value of exactly $-1$ is representable twice as the clamped result of the extreme negative integer. For normals this is negligible. For positions encoded as normalized signed values it means the bias term must be computed consistently with the convention, or the decoded model is shifted by one step at one end of its range — an error invisible on inspection and detectable only by comparing bounding boxes before and after a round trip.

An alternative that avoids the asymmetry entirely is to store positions as unsigned integers with an explicit per-primitive scale and offset carried in the mesh metadata, which is what glTF’s KHR_mesh_quantization extension enables. The accessor then holds unsigned values in $[0, 2^n - 1]$ and the offset moves the model into that box. The cost is one extra float per axis in the metadata and the requirement that every consumer apply it.

Normals: quantization plus renormalization

Normals are a different problem because they are constrained to the unit sphere. Quantizing three components independently against a per-mesh bounding box wastes most of the code space: a sphere’s axis-aligned bounding box is a cube, and the corners of that cube are far outside the sphere, so roughly half the representable triples describe non-unit vectors. Three 8-bit components give 16.7 million codes, of which only about 260 000 lie near the sphere’s surface at useful precision.

The practical response is not to store three components at all. Two common schemes:

Octahedral encoding maps the sphere onto an octahedron, then folds it into a square, and stores two components. It is a bijection everywhere except at the fold seam, has near-uniform error distribution, and its decode is a handful of operations including one abs and a conditional. With two 16-bit components it gives angular resolution on the order of a hundredth of a degree in the dense regions; with two 8-bit components the error is a few degrees, visible on smooth surfaces as faceting in specular highlights.

Store two components and reconstruct the third in the shader as $z = \sqrt{1 - x^2 - y^2}$, with the sign of $z$ implied by the vertex’s orientation. This works for normals in a hemisphere — the standard case for a mesh where every normal points away from the surface — and costs the square root plus a clamp to keep the radicand non-negative.

Either way the decoded vector is not exactly unit length. Quantization moves each component by up to half a step, and the reconstruction amplifies error where the radicand is near zero. Every consumer must renormalize after fetch. A shader that skips the renormalization produces normals whose length varies across the surface, which shows up in a lighting model as brightness variation unrelated to the geometry — a smooth object with a faint mottling that follows the quantization grid.

/* Decode an octahedral-encoded normal from two normalized shorts.
   The fold maps the lower half of the octahedron back onto the square. */
vec3 oct_decode(vec2 e) {
    vec3 n = vec3(e, 1.0 - abs(e.x) - abs(e.y));
    if (n.z < 0.0)
        n.xy = (1.0 - abs(n.yx)) * sign(n.xy);
    return normalize(n);            /* quantization is not length-preserving */
}

Tangents and the basis that must survive

A tangent is stored as a VEC4 because the fourth component carries the sign of the bitangent, which is needed when a mesh’s UV parameterization is mirrored on one side. Quantizing a tangent as three independent components is wrong in a specific way: the tangent is perpendicular to the normal, and independent quantization breaks that orthogonality by an amount proportional to the step size. A tangent basis that is not orthogonal produces a normal map that appears to shear, and the shearing direction follows the quantization grid rather than the surface.

The standard fix is to store the tangent in a frame relative to the normal, stored as two angles or two components in the normal’s tangent plane, and reconstruct the full vector by Gram-Schmidt or by direct construction from the normal and the two stored components. This makes the orthogonality exact up to the reconstruction’s own precision, and it reduces storage from four components to two plus the sign bit. The cost is that the decoder needs the normal, so the tangent cannot be fetched independently.

A cheaper alternative that many pipelines accept: store the tangent quantized and re-orthogonalize against the normal in the shader with one Gram-Schmidt step. That costs a dot product and a fused multiply-add per vertex and leaves a residual error of the same order as the quantization itself, which is acceptable for detail normal maps and marginal for high-frequency ones.

Texture coordinates are the exception

UVs do not quantize well, for a reason that is structural rather than numerical. UVs are frequently outside $[0, 1]$ because of tiling: a floor that repeats a texture twenty times has $u$ values up to 20. A per-mesh bound computed over those values is dominated by the tiling extent, so the quantization step in UV space becomes large, and since the texture is sampled at a resolution fixed in texels, the error translates directly into a positional error in the texture lookup.

The error is not uniform, either. UV precision needs to be high enough that adjacent vertices’ interpolated coordinates do not collide in texel space, which requires a step smaller than $1/W$ for a texture $W$ texels wide — for a 4096-wide texture, about $2.4 \times 10^{-4}$. With 16-bit unsigned storage over a $[0, 20]$ range, the step is $20/65535 \approx 3 \times 10^{-4}$, which is marginal; at 8 bits it is $0.078$, which is catastrophic.

The resolution is to quantize UVs after the atlas layout has fixed them into $[0, 1]$, not before, and to store the tiling factor separately as a per-material uniform rather than baking repetition into the vertex data. That decouples the range from the precision. It also means UV quantization belongs after the packing step described in UV chart packing and gutter space, because the packer’s output range is what the quantizer sees.

What the decoder pays, and where it belongs

Quantization moves work from the memory system to the arithmetic units. A 16-bit position fetch reads half the bytes of a float fetch, which matters when vertex fetch is bandwidth-bound; a mesh with 56 bytes per vertex at a million vertices reads 56 MB per full pass over the vertex buffer, and halving that is a real saving on any hardware. Against that, the decoder must apply a multiply-add per component, and the vertex shader’s arithmetic budget is finite.

Three placements are possible and they differ in cost.

Decoding in a compute pre-pass expands to floats once, writes them back, and lets the rasterizer read floats. This is the cheapest per-vertex path and the most expensive in memory: it doubles the vertex buffer’s footprint and requires a dispatch before each draw whose output changes.

Decoding in the vertex shader keeps storage small and pays the arithmetic per vertex per draw. For a mesh drawn many times per frame, as in a shadow pass plus a main pass plus a reflection pass, the arithmetic is paid repeatedly while the storage saving is paid once.

Decoding in a custom fetch path, where a small program reads the packed buffer and feeds the vertex shader directly, is the middle option available on APIs that expose it, and its benefit is that the expansion happens in the fetch unit rather than in the shader’s instruction stream.

The choice interacts with caching. A quantized vertex buffer of half the size means twice as many vertices per cache line, so a fetch pattern that walks vertices sequentially touches half as many lines — but if the vertex shader decodes on every pass, the arithmetic cost is multiplied by the pass count. For a mesh rendered in three passes, the crossover is roughly where the decode arithmetic exceeds the cost of the extra memory traffic, which is why storage-quantized formats tend to be used on meshes with high vertex counts and few passes, and expanded formats on meshes drawn repeatedly.

Failure modes that a validator will not catch

The bounds are the whole contract, and every failure traces back to them.

A mesh whose bounds are computed from the un-quantized data but whose stored values are quantized against different bounds decodes to a model of the wrong size, and if the mismatch is small it appears as a model that is very slightly too large or too small relative to its neighbours — visible only where two meshes touch. A pipeline should recompute the bounds from the quantized values after encoding and store those, so that the metadata describes what is actually on disk.

Degenerate bounds break the formula outright: a mesh that is planar in one axis has $\text{max} = \text{min}$, so the step is zero and the division by the range is undefined. The correct handling is to store a step of one (or any fixed nonzero value) for that axis and let the decoded value be the constant, since every vertex shares it. Encoders that do not special-case this produce NaNs for perfectly flat geometry, which is common — a ground plane, a decal quad, a UI mesh.

Extreme values also break the arithmetic. A mesh with a single vertex at a huge coordinate, such as a stray vertex at $10^{30}$ from a modelling accident, sets the bounds to that value, and every other vertex quantizes to a step of $10^{25}$, collapsing the model to a few grid points. Clipping the bounds to a high percentile, or rejecting outliers before computing them, is the mitigation; the failure without it looks like a model that vanished.

Finally, the quantization of positions interacts with the handedness and unit conversions of an import stage: the same model in different units has different bounds and therefore different absolute error. That dependency is why the conditioning order in a pipeline matters, and it is the reason axis conventions and unit scale should be resolved before any quantization step. The same is true of the vertex layout itself, since a quantized attribute’s offset within an interleaved record must still satisfy the alignment rules described in glTF buffer views and accessor strides.