Optimizations That Move Cost Instead of Removing It

Most long-lived optimizations do not delete cost. They relocate it: from the read path to the write path, from the median to the tail, from the server to the client, from the present to the future, from the machine to the operator. The relocation is frequently worth doing, because the metric being optimized is the one that matters and the metric receiving the cost is not tightly constrained. The failure mode is not the trade; it is treating the trade as a free win because only one side of it was instrumented.

A cost-shift is identifiable in advance because every relocation has a destination that can be named. Naming it converts an uncontrolled side effect into an explicit decision with an owner.

Batching and the latency cost of amortization

Batching is the canonical case. The per-operation cost of an I/O or RPC is largely fixed — a syscall, a packet header, a lock acquisition, a scheduling decision — so performing $n$ operations in one batch reduces the per-operation cost by a factor approaching $n$ in the limit where the fixed cost dominates.

The cost appears as waiting. A batch of $n$ is only worth forming if its members are available, so the first member of a batch waits for the arrival of the $n$th. Under a fixed arrival rate $\lambda$, the expected wait is on the order of $n/(2\lambda)$ for uniformly arriving members. Increasing the batch size quadratically improves throughput per fixed cost and linearly increases the latency floor. At the extreme, batching converts a system with a fixed latency profile into one with a throughput ceiling and a latency floor set by the batch window, and the floor is paid by every request including the ones that would have been fast.

The choice of $n$ is therefore not a tuning parameter but a statement about which metric the system optimizes, in the sense that fixing an objective fixes the exchange rate between the two. Where the batch is formed per caller (as in a group commit), the caller’s own latency is unaffected and the trade is different: the cost is paid in concurrency, since a caller must wait for other callers. Where the batch is formed by a timer, the cost is a fixed added delay on every operation, which is easiest to account for and hardest to justify for latency-sensitive traffic.

Precomputation and the invalidation bill

Precomputation moves work from query time to write time or to a scheduled job. A materialized view, an index, a denormalized counter, or a precomputed aggregate all buy read latency with write cost and storage.

The read path is easy to measure and the write path is usually less instrumented, so the trade is recorded as a win on the evidence available. The hidden costs are three.

The first is write amplification: each write must update the derived structures, so the cost of a write grows in proportion to the number of derived forms it maintains. A table with five indexes costs five index updates per insert, and the cost is proportional to the indexed columns’ size, so an index over a wide column is far more expensive than an index over an integer.

The second is staleness. A derived value is correct only as of its last update, and the gap between that and the read is the staleness window. Shortening the window by making updates synchronous moves the cost back to the write path; lengthening it is a correctness change that must be decided by whoever consumes the value, not by whoever optimizes the read.

The third is rebuild cost. Any derived structure must be reconstructible after a corruption or a schema change, and the rebuild consumes the same capacity the system would otherwise serve with. A precomputation that is cheap in steady state and impossible to rebuild within the maintenance window is a liability that appears only during an incident.

Compression and the bandwidth-versus-CPU exchange

Compression trades transfer or storage volume against CPU. The ratio is not a single number; it depends on the compressor, the input’s entropy, and the level chosen. The relevant comparison is not compressed size but time: the encode plus transmit plus decode time against the transmit time of the uncompressed data.

For a fast compressor the encode and decode costs are a few hundred megabytes per second per core and the ratio is typically a factor of two to three on ordinary text and structured data. Whether that is a win depends entirely on the bandwidth of the medium. On a network link whose throughput per transfer is a few tens of megabytes per second, a threefold reduction is decisive. On a link that is ten times faster than the CPU can compress, compression makes the transfer slower. The same code, the same setting, opposite conclusions, because the ratio being optimized changed.

Compression also shifts cost between the two ends and changes the variance. A compressed transfer is more sensitive to the loss of a CPU core than an uncompressed one, and a decode stall is a different failure than a bandwidth stall: the former is measurable as CPU time in a profile and the latter as waiting. Diagnosing the wrong one leads to optimizing the wrong tier.

Caching, pool sharing, and the shift onto future traffic

Two further forms of cost shift deserve naming because they are often invisible in the measurement window.

A cache moves work from the cache-hit path to the miss path and to the invalidation path, and it also moves future work to the present: the entry is built now to be used later, and if it is not used the construction was wasted. The break-even arithmetic in the hit-rate threshold prices the first two; the third is a property of the expiry policy and the access distribution, and it shows up as a cache that consumes CPU proportional to traffic without reducing anything, because every entry is computed and discarded.

Connection or buffer pooling moves the cost of establishing a resource from the request to the pool’s lifecycle. The request path gets cheaper; the pool’s growth, shrinking, health checks, and reconnection storms become costs with their own dynamics. A pool that is sized for peak demand holds resources that are idle at other times, which is a real cost to everything else sharing the machine; a pool that shrinks aggressively reintroduces the establishment cost exactly when load returns and simultaneously, since load returns to many callers at once.

In both cases the shift is onto a time horizon longer than the measurement window. An evaluation run of a few minutes at steady state cannot see a cost that accrues over hours, so the accounting has to be done analytically or over a window long enough for the accrual to appear.

Accounting for the shift before committing

Four steps make the trade explicit and disable most of the failure modes.

State the metric being optimized and the metric receiving the cost, with units. “Requests per second” receiving “p99 latency in milliseconds” is a comparable pair; “performance” receiving “performance” is not.

Quantify both sides with a model, not a measurement of one side. The transfer of work is usually computable from the batch size, the amplification factor, or the bandwidth ratio; the constants need measurement but the direction does not.

Check the receiving metric against its own constraint. A system with a latency budget and no throughput problem should not adopt a throughput optimization that spends latency; a system whose write path is already the constraint should not adopt a read optimization paid for in writes.

Decide who owns the receiving side. A precomputation whose staleness is consumed by a product feature has shifted a performance decision into a product decision, and the owner of that feature has to be part of the choice. Where no owner exists, the cost is unconstrained and will grow.

The last point is the reason cost shifts persist. The optimization is visible in the metric that has an owner and a dashboard; the cost is visible only to the metric that has neither. Building the second dashboard is a prerequisite for evaluating the first, and is the least glamorous part of deciding when to stop improving a system that has been optimized by relocation many times over.