Inference Systems · Cluster
Inference Economics
Cost per token and capacity planning.
- 01 Prefill and Decode: The Two Cost Curves Behind One Token Price Why the same model spends prefill time on arithmetic and decode time on memory bandwidth, and how that split decides what a token costs to serve.
- 02 The KV Cache Memory Budget and What It Costs per Token Deriving bytes per cached token from attention shapes, then turning the freed capacity into a hard ceiling on concurrency and a floor on cost per token.
- 03 Batch Size, Occupancy, and the Marginal Cost of a Token How the cost of one more concurrent request falls as the batch grows, where it stops falling, and why the knee of that curve sets the price floor.
- 04 Sizing a Replica Pool from Arrival Rate and Latency Target Queueing arithmetic that converts requests per second and a latency objective into a number of replicas, including the headroom that utilization hides.
- 05 Prompt Cache Economics: Hit Rate, Prefix Length, and Break-Even What a cache hit actually saves, how prefix granularity bounds the achievable hit rate, and where cache storage competes with concurrency for the same memory.
- 06 Speculative Decoding Cost Accounting: The Acceptance Rate Threshold How draft tokens turn a bandwidth-bound decode step into several tokens, the acceptance rate at which the extra compute pays for itself, and why batch size moves that threshold.
- 07 Reserved Versus On-Demand Capacity: The Break-Even Utilization Comparing a discounted commitment against pay-per-use pricing when demand has a diurnal shape, including the utilization at which the commitment stops saving money.
- 08 Why Token Billing Diverges from GPU-Seconds The gap between what a request costs in hardware time and what it costs at a per-token price, and the workload shapes that make the two accounting systems disagree.
- 09 Amortized Cost per Token: Depreciation, Power, and Idle Capacity Building a total cost of ownership for a token service from hardware life, power draw, and utilization, and showing why idle capacity is the largest term.
- 10 Autoscaling a Token Service with Cold Starts and Diurnal Load Why reactive scaling fails for inference replicas, how to derive the lead time a scale-out needs, and what a warm pool costs against the peaks it covers.