Tag
quantization
- Sep 24, 2026 Rate-Distortion Theory and the Shape of Lossy Compression Why lossy coding is governed by a curve rather than a single number, how the Gaussian bound is derived, and why vector quantization beats scalar quantization at the same rate. computation theory
- May 25, 2026 Blockwise 8-Bit Optimizer State Without Losing the Second Moment Quantising Adam moments to eight bits cuts state memory by four, but small second-moment values quantise to zero. Blockwise scaling and dynamic exponent sharing fix that. deep learning
- Apr 21, 2026 Banding, Bit Depth, and the Trade-offs of Error Diffusion Dither Why quantization on an 8-bit gradient shows contours, how the eye's contrast sensitivity makes them visible, and what ordered and error-diffusion dither cost. computer graphics
- Feb 6, 2026 Coarse Quantization with Inverted Files and the nprobe Setting How an inverted-file index reduces a full scan to a few cluster scans, what the centroid training step optimizes, and why the nprobe parameter buys recall at a nearly linear price in work. information retrieval
- Jan 8, 2026 Approximate Nearest Neighbour Indexes and the Recall They Trade Away How graph, cluster, and hash indexes reduce a linear scan to a bounded walk, which knob controls recall, and why recall is a property of the query distribution rather than the index. inference systems
- Dec 4, 2025 Keyframe Reduction and the Error Metric That Decides It How a dense sampled curve becomes a sparse key list, why the error must be measured against the reconstructed curve and not the original samples, and where per-channel reduction breaks a skeleton. computer graphics
- Aug 6, 2025 Vertex Attribute Quantization and the Range It Depends On How positions and normals become 16-bit integers with a per-mesh scale and bias, where the error lands, and why renormalization after fetch is mandatory. computer graphics
- Sep 25, 2024 Product Quantization and the Asymmetric Distance Computation How splitting a vector into subspaces and coding each against a small codebook replaces a distance computation with table lookups, and where the reconstruction error shows up in the ranking. information retrieval
- May 6, 2024 Quantization Step Size, the Six-Decibel Rule, and Dither Why each added bit buys roughly six decibels of signal-to-noise ratio, when the uniform-error model stops holding, and what dither trades away to fix it. scientific computing
- Dec 21, 2023 Perceptually Uniform Colormaps and the Failure of Rainbow Ramps How a scalar field becomes color through a lookup table, why monotonic lightness and equal perceptual steps are the two constraints that matter, and where banding and false contours come from. scientific computing
- Apr 10, 2023 The sRGB Transfer Function: Piecewise Curve, Constants, and Decode Cost How the sRGB encoding is defined, why it bends into a straight line near black, and what the exact thresholds cost in a shader or a texture lookup table. computer graphics
- May 14, 2022 Compressed BVH Nodes and Quantized Bounds How bounds shrink to byte and short quanta, what the inverse mapping costs per test, and why the compressed tree changes traversal arithmetic and memory traffic. computer graphics