Tag
optimization
- Feb 27, 2026 Decoupled Weight Decay and Adaptive Scaling Why adding a shrinkage term to the gradient is not the same as shrinking the parameter, how the adaptive denominator changes the effective decay per coordinate, and what the decoupled update fixes. deep learning
- Aug 1, 2025 Undefined Behaviour as a Compiler Permission Why the optimiser treats undefined behaviour as a promise about what never happens, and how that promise lets it delete checks, fold comparisons, and reorder loads. systems programming
- Mar 22, 2025 Debug Information for Optimized Code: Line Tables, Location Lists, and Lost Variables At -O2 the compiler moves, merges, and deletes variables. DWARF describes the result with location lists and inline scopes, and the debugger reconstructs values that no longer live anywhere. compilers
- Feb 11, 2025 Gradient Clipping by Norm and Loss-Scale Rescaling What a per-tensor global norm clip bounds, how the scale factor is computed and applied, and why clipping interacts with the loss scale used in low-precision training. deep learning
- Dec 1, 2024 Gradient Noise Scale and Critical Batch Size How the ratio of gradient variance to gradient squared norm sets the batch size beyond which additional examples buy no reduction in steps, and why that ratio rises over training. deep learning
- Oct 4, 2024 Cosine and Linear Decay Schedules as Step-Size Integrals What a learning rate schedule controls beyond the final value, how the shape distributes total parameter displacement across training, and why the choice between linear and cosine changes little at equal area. deep learning
- Sep 14, 2024 Aliasing Rules for References and the Role of UnsafeCell A shared reference is a promise of immutability that the optimiser acts on, and UnsafeCell is the single opt-out that makes interior mutability representable without lying to the compiler. systems programming
- Aug 12, 2024 Strict Aliasing and the Cases Where Type Punning Is Legal The effective type rules permit far more access through differing types than the folklore suggests, and the sanctioned escape hatches are narrower than the compiler flags that disable the analysis. systems programming
- Feb 17, 2024 Learning Rate Warmup and the Early Instability Window Why the first few hundred optimization steps are different from the rest, what large early updates do to adaptive optimizers, and how warmup bounds the damage. deep learning
- Oct 4, 2023 Object Shapes, Inline Caches, and the Cost of a Polymorphic Site Why property access is fast when objects share a hidden class, how shapes transition, and what turns an inline cache from a lookup into a hash probe. web platform
- Jun 19, 2023 How Escape Analysis Removes a Heap Allocation How a compiler proves that an allocation cannot be observed outside its function, and what it must give up when object identity or a stored pointer blocks the proof. programming languages