Inference Systems · Inference Economics
Amortized Cost per Token: Depreciation, Power, and Idle Capacity
Building a total cost of ownership for a token service from hardware life, power draw, and utilization, and showing why idle capacity is the largest term.
The cost of producing one token is not the price of the hardware divided by the tokens the hardware produces. That quotient leaves out the years in which the hardware produces nothing, the power it draws while waiting, the networking and storage that surround it, and the failure rate that requires spares. Adding those terms changes the number by a factor, and more importantly it changes which decisions matter: at realistic utilizations, the dominant lever is not the accelerator’s throughput but the fraction of the day it is used.
The four terms
A defensible cost per token starts from an hourly cost of ownership for one replica and divides by the tokens that replica delivers in that hour.
The first term is depreciation: the purchase price of the accelerator and its host, spread over the service life. A three-year life is a common convention for accelerators that are superseded on roughly that cadence, and it is a modelling choice rather than a physical constant. A device costing tens of thousands of dollars over three years, with 8,760 hours in a year, contributes a few dollars per hour before anything else.
The second term is power. The accelerator’s draw under load is substantial, the host and its fans add to it, and the facility overhead multiplies both by a power usage effectiveness factor that accounts for cooling and distribution. A rack of accelerators draws enough power that the electricity bill is not a rounding error against depreciation, and the ratio between the two depends on local energy prices, which vary by a large factor between regions.
The third term is the surrounding infrastructure: the network fabric, the storage for model weights and logs, the load balancers, and the engineering time to operate it. These are shared across replicas and are usually expressed as a multiplier on the per-replica cost rather than as a separate line.
The fourth term is the one that is usually omitted and usually decisive: the capacity that exists but is not producing tokens. Idle replicas depreciate and draw power exactly as fast as busy ones, so the cost per token scales as the inverse of utilization.
$$\text{cost per token} \approx \frac{C_{\text{dep}} + C_{\text{power}} + C_{\text{infra}}}{u \cdot 3600 \cdot \text{tokens per second at full load}}$$
The utilization $u$ in the denominator is the fraction of hours in which the replica is doing useful work. It is not the fraction of the hour in which the accelerator is busy, which is a different and much higher number, because a replica that is serving a single request at batch 1 is busy and delivering very few tokens.
Two utilizations, and only one of them appears in the price
The distinction between temporal utilization and batch occupancy is where most cost models lose accuracy. A replica that is resident for 24 hours and serves requests for 20 of them has a temporal utilization of 0.83. If those requests are served at a batch of 4 when the replica could hold 64, the effective utilization of its capacity is 0.83 times 4/64, or about 0.05. The cost per token differs by a factor of sixteen between those two readings, and only the second reflects the hardware’s actual productivity.
The correct denominator is therefore tokens actually delivered per hour, not a nominal throughput multiplied by a temporal fraction. That number can be measured directly and it absorbs both effects. The cost model’s job is then to explain why the measured number is what it is and which term would move it.
For a decode-heavy service, tokens per hour is bounded by the batch economics: tokens per step times steps per hour. Steps per hour is bounded by the step time, which is bounded by memory bandwidth and by the latency target. At a batch of $b$ and a step time $t$, tokens per hour is $3600 b / t$, and the step time grows with $b$ only past the compute crossover. So the achievable tokens per hour at a fixed latency target is a property of the model’s bandwidth requirements and the hardware’s bandwidth, and the cost per token is the hourly cost divided by it.
For a prefill-heavy service the bound is different: tokens per hour is flops per hour divided by flops per token, so it is close to the peak arithmetic throughput, and the utilization that matters is how much of the hour carries prefill work at all. Prompt-heavy services are therefore more sensitive to temporal utilization, and output-heavy services to batch occupancy, a split that follows directly from the two cost curves behind any single token price.
Power is not proportional to work
An accelerator at idle still draws a substantial fraction of its loaded power, because the memory system and the leakage current do not switch off. The consequence for cost accounting is that power per token is worse than linear in utilization: the idle draw is charged to the tokens that are produced during the busy period, so halving utilization more than doubles the power cost per token. This is the reason that facility-level efficiency arguments center on keeping accelerators busy rather than on making each operation cheaper.
It also means that a plan which spreads load across more replicas than necessary pays twice: once for the depreciation of the idle replicas and once for their idle power. Consolidating the same work onto fewer, busier replicas reduces both terms, subject to the latency target and to the failure-domain requirement that no single replica be a single point of failure for a critical service.
Life and resale change the depreciation term
A straight-line three-year life is a convention that ignores two facts. Accelerators retain resale value, so the net cost is the purchase price minus the residual, and the residual falls with age at a rate that is not linear. And the useful life is set by the workload, not by the calendar: an accelerator that lacks the memory capacity for a newer model’s weights becomes unusable for that model before it wears out, which shortens its economic life for that purpose while leaving it useful for smaller models.
The practical implication is that a cost model should be run at more than one assumed life. At a three-year life the depreciation term is a third of the price per year; at five years it is a fifth, and the same hardware looks substantially cheaper per token. The choice of life is a statement about how long the workload will remain compatible with the hardware, and it should be justified by the model sizes and precisions the service expects to run rather than by an accounting default.
The same reasoning applies to the decision between owning and renting. Renting converts depreciation and residual-value risk into an hourly rate, at a premium that reflects the provider’s own cost of capital and utilization. The comparison is the reserved-versus-on-demand calculation with the depreciation term made explicit, and it comes out differently depending on whether the workload’s hardware requirements are stable over the term.
Where the money actually goes
The table below is a structure, not a set of measurements. The point is the shape: at low utilization the idle term dominates, and at high utilization the hardware terms dominate in proportion to their share of the hourly cost.
| Term | Scales with | Reduced by |
|---|---|---|
| Depreciation | calendar time, not load | longer economic life, higher resale, cheaper hardware per unit of bandwidth |
| Power | time at idle plus time at load | consolidation, higher utilization, efficient facilities |
| Infrastructure | replica count | sharing fabric across more replicas, fewer larger replicas |
| Idle capacity | inverse of utilization | batching, routing, elastic capacity, workload shaping |
| Rework | wasted tokens, retries, preemptions | admission control, cache reuse, fewer restarts |
The rework term deserves its own line because it is invisible in a cost model that counts only successful tokens. A request that is preempted mid-generation and restarted consumes accelerator time twice and produces one output. A retry after a timeout does the same. A response truncated by an output limit and regenerated with a longer limit does it three times. Each of these is a real cost that no per-token price reflects, and their frequency rises exactly when the system is under pressure, which is when the capacity plan is already strained.
The number that belongs in a decision
A single cost per token is a summary, and summaries hide the levers. The useful form is a cost per token decomposed into its terms at the measured utilization, with the sensitivity of each term to the decisions under consideration. Raising the batch from 8 to 16 changes the throughput term; shortening the mean context changes the memory term and the throughput term together; moving from on-demand to committed capacity changes the depreciation term and the idle term together. A plan that reports only the total cannot distinguish these, and will tend to reach for the lever that is easiest to describe rather than the one that is largest.
The measurement that keeps the model honest is accelerator-hours per million tokens, tracked alongside the token volume and the request mix. It rises when utilization falls, when the mix shifts toward expensive requests, and when rework increases; it falls when batching improves or a cache absorbs prefill. Those are precisely the phenomena the decomposition is meant to explain, and the ratio is available without any of the modelling assumptions that the decomposition requires.
References
- The Datacenter as a Computer: Designing Warehouse-Scale Machines — Barroso, Clidaras & Holzle, Morgan & Claypool, 2013
- Above the Clouds: A Berkeley View of Cloud Computing — Armbrust et al., UC Berkeley, 2009
- Efficiently Scaling Transformer Inference — Pope et al., MLSys 2023