Choosing an Objective Function for a Mixed Workload

Optimization needs a scalar. Real systems produce several, and the choice among them is not a reporting detail bolted on at the end: it determines which candidates look attractive, which measurements are worth taking, and whether a change that lands is later judged a success.

The clearest way to see the coupling is to consider two candidate changes and two objectives. One change shaves fifty milliseconds from the common path and nothing from the slow path. The other shaves three hundred milliseconds from a path taken by five percent of requests and nothing from the rest. Ranked by mean latency they differ by a factor; ranked by 99th percentile only the second one exists. Neither ranking is wrong. The objective chose the winner.

Averages over heterogeneous populations

A mean is the sum of durations divided by the count, which makes it a weighted average in which expensive requests dominate in proportion to their cost. This property has two consequences that pull in opposite directions.

It makes the mean sensitive to a minority of pathological requests, which is often what an operator wants: a long-tail of retries or timeouts is a symptom, and the mean moves when it appears. It also makes the mean insensitive to a change that helps many requests a little, because the arithmetic of an average charges each request in proportion to its cost, not in proportion to its count.

The stronger objection to a mean is that a mean of a skewed distribution describes a value that no request experiences. Latency distributions are right-skewed, with a long upper tail; the mean sits somewhere in the region where the density is falling, below the slow requests and above the bulk. An optimization that moves the body of the distribution closer to the mean and an optimization that truncates the tail can produce indistinguishable means while doing entirely different things.

Weighting by request count rather than cost — the geometric mean over per-request ratios, or the arithmetic mean of per-class means — answers the “many requests, small improvement” question. It is the natural objective when the goal is a uniform experience and the traffic is a mix of cheap and expensive operations that matter equally.

Percentiles and the responsibility for the tail

A percentile is a threshold: the value $x$ such that a proportion $p$ of observations are at or below $x$. Choosing $p$ is choosing how many requests to ignore.

The choice is constrained by arithmetic. At $p = 0.99$ a system serving one million requests per hour produces ten thousand requests above the threshold per hour; at $p = 0.9999$ it produces a hundred. The relevant $p$ is set by the rate at which tail events are tolerable, not by convention, and a percentile reported without the request rate is missing the quantity that makes it interpretable.

Percentiles are also not additive across a mix. The 99th percentile of a combined population is not a function of the constituent percentiles, so a system monitored by merging per-service p99 values is reporting something that does not correspond to the end-to-end p99. Where the tail is the objective, the measurement must be taken at the point where the objective is defined; a per-stage percentile cannot be composed into an end-to-end one. This is the reason tail latency work emphasizes tracing complete requests rather than aggregating stage statistics.

A further property makes tails expensive to improve: the tail of a composite system is dominated by the slowest of many parallel or sequential stages, and the tail of each stage is not rare in the composite. With a hundred sequential stages, a stage-level operation in the top one percent occurs in roughly sixty-three percent of requests. Coordinates that look acceptable individually produce a composite tail well beyond what any stage’s percentile suggests. The mitigation — hedged requests, tail-tolerant designs, redundant issuance — works by spending resources to remove the dependence on the outlier, which is a cost the objective must permit.

Goodput and the definition of useful work

Both mean and percentile latency implicitly assume each request is worth serving. A third family of objectives prices the outcome: goodput counts operations that completed within a deadline and returned a usable result, and divides by time. The index in “goodput” is doing real work, because it draws a line between a fast response and a fast response that was correct.

Three failure modes appear under goodput that are invisible under latency.

A deadline miss is not merely a slow response but a wasted one, and its resource consumption is charged against the system that failed to use it. A system that completes ninety percent of requests in ten milliseconds and ten percent in four hundred while the deadline is a hundred has a goodput of ninety percent of its throughput, and its wasted work is ten percent of the machine.

Caching and precomputation can raise the latency distribution while lowering goodput, if the cached response is stale beyond the tolerance the consumer needs. The staleness window is then a term in the objective, not a footnote.

Retries raise throughput and can lower goodput: a request that fails and is retried twice consumes three times the capacity of one that succeeds, and the third attempt is often the one that misses the deadline. An objective that counts attempts as work rewards the wrong behaviour.

Defining goodput requires, per operation class, a deadline and a correctness criterion. Both are policy decisions that must be recorded where the metric is computed; a goodput number whose deadline is not stated is an ordinal at best.

Mixing classes without letting one class dominate

Real workloads contain classes with different service level expectations: interactive reads, bulk exports, background maintenance, and administrative changes. Any single number over the union of these is dominated by whichever class is largest or most expensive.

Two standard resolutions exist, and they answer different questions.

The first is a per-class constraint: each class gets a threshold, and the objective is the count of classes meeting their thresholds, or a lexicographic ordering in which interactive latency must be met before throughput is considered. This matches how systems are actually operated and resists the arithmetic problem, because a class that is not represented in the mix cannot be averaged away. It has the drawback of being insensitive to large improvements within a class for a class already meeting its threshold.

The second is a weighted combination, with weights set from the cost of violating each class’s threshold. The weights are a business quantity and must be supplied rather than inferred; a weight chosen because it makes two candidate scores land on either side of an arbitrary boundary is a decision smuggled into a coefficient.

-- Goodput per class: completed within deadline and returned successfully.
SELECT op_class,
       COUNT(*) FILTER (
         WHERE status = 'ok' AND duration_ms <= deadline_ms
       )::float
       / NULLIF(COUNT(*), 0) AS goodput_ratio,
       COUNT(*) AS attempted
FROM request_log
WHERE started_at >= now() - interval '1 hour'
GROUP BY op_class
ORDER BY attempted DESC;

Two implementation details matter more than they appear. Percentiles and ratios computed from samples need the deadline to be stored with the row rather than applied later, because the deadline in force when the request was made is part of its contract. And the class must be recorded at the point of origination, since it cannot be recovered from the log afterwards.

What the objective does to the candidate list

Once fixed, the objective propagates into every number in this cluster. A cost share is measured against the chosen metric: a component that is five percent of CPU time and zero percent of p99 latency is a candidate only if the objective is throughput or capacity. The probability that a change survives becomes the probability that it helps the specified class, which differs sharply from the probability that it helps in general. A mean-latency objective will rate a change that helps the majority and slightly harms the tail as positive; a tail objective will reject the same change.

The practical consequence is that the objective should be written down before the profile is read, and should not be revised in response to which candidate it favours. Revising it after seeing the ranking turns a measurement process into a search for a metric that justifies a preference. The question of when a fixed objective has been satisfied to the point of diminishing return belongs to recognizing the optimization plateau; the question of what to do when no single candidate dominates under any objective belongs to Pareto frontiers in configuration tuning.