Tag
cost-model
- Jun 22, 2026 Prompt Cache Economics: Hit Rate, Prefix Length, and Break-Even What a cache hit actually saves, how prefix granularity bounds the achievable hit rate, and where cache storage competes with concurrency for the same memory. inference systems
- Jun 10, 2026 Costing a Parallel Plan: Workers, Gather, and the Serial Fraction Why dividing a plan cost by the number of workers overestimates the gain, how the planner prices worker startup and row transfer, and when a parallel plan loses to a serial one. databases
- Jun 5, 2026 What an Allreduce Actually Moves: Trees, Rings, and the Cost Model Why the obvious gather-reduce-scatter is the wrong shape, how ring and recursive-doubling algorithms trade latency for bandwidth, and where the two cross over. scientific computing
- Feb 4, 2026 Prefill and Decode: The Two Cost Curves Behind One Token Price Why the same model spends prefill time on arithmetic and decode time on memory bandwidth, and how that split decides what a token costs to serve. inference systems
- Nov 6, 2025 Reserved Versus On-Demand Capacity: The Break-Even Utilization Comparing a discounted commitment against pay-per-use pricing when demand has a diurnal shape, including the utilization at which the commitment stops saving money. inference systems
- Oct 30, 2025 Inlining Decisions and the Cost Model Behind Them Inlining is chosen by a profitability estimate over caller and callee size, call frequency, and constant-argument opportunities, and every model trades code size against a set of downstream optimizations. compilers
- Jul 29, 2025 Partitioned Boolean Quadratic Programming for Register Allocation How encoding allocation as per-node costs plus per-edge cost matrices lets a solver trade spills against copies with a single objective, and why reduction rules beat coloring heuristics on hard graphs. compilers
- Jun 12, 2025 Speculative Decoding Cost Accounting: The Acceptance Rate Threshold How draft tokens turn a bandwidth-bound decode step into several tokens, the acceptance rate at which the extra compute pays for itself, and why batch size moves that threshold. inference systems
- Jun 4, 2025 The Cost Tiers of a Single Allocation Request One allocation call can cost tens of cycles or tens of thousands depending on which tier it stops at: thread cache, shared free list, kernel mapping, or first write. performance engineering
- May 13, 2025 Batch Size, Occupancy, and the Marginal Cost of a Token How the cost of one more concurrent request falls as the batch grows, where it stops falling, and why the knee of that curve sets the price floor. inference systems
- Dec 4, 2024 How a Query Cost Model Reduces I/O and CPU to One Number Why planner costs are weighted units rather than milliseconds, how sequential and random page reads enter the sum, and which effects the arithmetic deliberately ignores. databases
- Nov 13, 2024 The Crossover Between an Index Scan and a Sequential Scan Why fetching a few percent of a table by index can cost more than reading all of it, how physical correlation moves the crossover point, and what bitmap scans change. databases
- Feb 20, 2024 The KV Cache Memory Budget and What It Costs per Token Deriving bytes per cached token from attention shapes, then turning the freed capacity into a hard ceiling on concurrency and a floor on cost per token. inference systems
- Dec 20, 2023 Unit-Cost RAM, Word RAM and Bit Complexity as Three Different Machines The same algorithm can be linear, logarithmic or quadratic depending on whether operations are charged per step, per word, or per bit, and the choice of model decides which bounds are comparable. algorithms
- Jul 14, 2023 Amortized Cost per Token: Depreciation, Power, and Idle Capacity Building a total cost of ownership for a token service from hardware life, power draw, and utilization, and showing why idle capacity is the largest term. inference systems
- Apr 5, 2023 Sort, Hash, and Materialize: Memory Limits and Spill to Disk How a hash join decides it must partition, what a sort does when the work memory runs out, and why an underestimated row count becomes a disk-bound plan. databases
- Dec 13, 2022 Sizing the Window: Why More Entries Stop Paying The diminishing returns of a larger instruction window, the queueing model that explains them, and the second-order effects that make big windows expensive. computer architecture
- Jun 27, 2022 Why Token Billing Diverges from GPU-Seconds The gap between what a request costs in hardware time and what it costs at a per-token price, and the workload shapes that make the two accounting systems disagree. inference systems