Pareto Frontiers and Cost-Benefit Ranking of Architecture Options

An architecture choice with four candidate designs, three quality scenarios, and a budget produces an argument rather than an answer, because the criteria are in different units and every participant weights them differently. The usual response is a weighted score, which requires somebody to declare that one millisecond of latency is worth a fixed amount of money — a claim nobody can defend and everybody can contest. Pareto dominance postpones that claim. It removes the options that no weighting could prefer, and leaves a small set where the argument is genuinely about values rather than about arithmetic.

Dominance before weighting

An option A dominates option B when A is at least as good on every criterion and strictly better on at least one. A dominated option cannot win under any set of non-negative weights, because there is no weighting under which its deficits are compensated by its strengths; it is simply worse. Removing dominated options is therefore free of value judgements, and it is the cheapest operation in the whole analysis.

What remains is the Pareto frontier, the set of options where improving any criterion requires giving something up. In two dimensions the frontier is the staircase-shaped upper-left boundary of a scatter plot, and it can be read directly. In more dimensions the geometry disappears but the definition does not, and the size of the frontier grows with the number of criteria: with many criteria, most options are non-dominated, which is the practical limit of the technique.

Four criteria is about the useful maximum for a hand-built frontier. Beyond that the dominance filter stops eliminating, and the remaining comparison needs a different tool, usually a cost-benefit ratio or an explicit weighting.

Getting measures onto comparable scales

Dominance needs only an ordering per criterion, which is why it works with mixed units: milliseconds, currency, incident count, engineer-days. Comparing options on the frontier, however, needs magnitudes, and magnitudes require a common scale.

The standard transformation is a desirability function per criterion, mapping the raw measure onto [0, 1] with a stated shape. Three shapes cover most architectural criteria:

Linear between bounds. A scenario asks for a p99 under 300 ms; below 100 ms is no better in practice and above 600 ms is unusable. Mapping that interval linearly gives a defensible curve and forces the bounds to be stated. The bounds are a judgement, which is why they belong in the scenario’s measure rather than in the scoring spreadsheet.

Threshold. A scenario that requires no data loss is not partially satisfied. Binary criteria collapse to 0 or 1 and should not be blended into a score, since an option that fails a must-have is not a worse option, it is not an option. Filter on those first, then score.

Saturating. More of a good thing stops helping: an availability figure above 99.99 percent may not change any behaviour that the business can distinguish. A saturating curve prevents the analysis from rewarding over-engineering on one criterion at the expense of all others.

The scale matters as much as the shape. Linear scaling between arbitrary bounds makes the result an artefact of where the bounds were placed, which is why the bounds need to come from the stakeholder conversation rather than from the range observed in the data. Ranking options by the spread of the sample is how a frontier becomes a mirror of the sample.

Comparing four options

Consider a design choice for a reporting subsystem, evaluated against four criteria: p99 query latency, cost of the monthly infrastructure bill, time to add a new report type, and operational burden. The numbers are illustrative of the arithmetic, not measurements of any system.

Option	p99 latency	Monthly cost	New report effort	Operational burden
A: one database, read replica	800 ms	low	2 days	low
B: precomputed aggregates	120 ms	medium	6 days	medium
C: columnar store, separate	90 ms	high	3 days	high
D: columnar store, one oversized node	150 ms	high	8 days	high

Option A is not dominated: it is cheaper, quicker to extend, and easier to run than any alternative, though it is the slowest. B is also undominated: it is faster than A, cheaper and easier to run than C, but slower to extend than either. C has the best latency, at high cost and high operational burden. D is dominated by C — slower, slower to extend, and no cheaper or easier to run — so it can be dropped without discussion.

Three options survive: A, B, C. The elimination of D is the entire value of the first pass. It takes one person ten minutes, requires no weights, and removes an option somebody had been advocating.

def dominates(a, b, maximise):
    """a dominates b: no worse on every criterion, strictly better on one."""
    better = False
    for key, is_max in maximise.items():
        va, vb = a[key], b[key]
        if (va < vb) if is_max else (va > vb):
            return False            # a is worse on this criterion
        if va != vb:
            better = True
    return better

def frontier(options, maximise):
    return [o for o in options
            if not any(dominates(other, o, maximise)
                       for other in options if other is not o)]

The maximise dictionary carries the direction per criterion, which is where most hand-rolled comparisons go wrong: a lower latency is better, a lower cost is better, a lower effort is better, and a lower operational burden is better, so in this example every criterion is minimised. Mixing directions in one score without normalising produces an ordering that rewards whichever criterion happens to have the largest raw numbers.

Cost-benefit ratio and what it hides

The Cost Benefit Analysis Method attaches a benefit score to each architecture and divides by cost. Its benefit side comes from the utility tree: stakeholders distribute 100 points across the scenarios, and each architecture receives a response level per scenario, normalised to a common range. The benefit is the weighted sum of response levels; the ratio is benefit divided by cost, and options are ranked by ratio.

The ratio answers a question worth asking — which option returns the most quality per unit of money — and it has three characteristic failure modes.

It rewards cheapness. An option that improves one scenario slightly at near-zero cost can outrank an option that fixes a high-importance scenario at moderate cost, because the denominator dominates. The mitigation is to compute the ratio only among options that pass the must-have filter, and to examine the top of the ranking rather than the whole list.

It discards marginal information. A ratio is an average over the whole option, and the decision-relevant quantity is often marginal: what does the next increment of quality cost? Two options with the same ratio can differ entirely in what an additional investment would buy, and the marginal rate on the frontier is where that shows.

It depends on cost estimates of wildly different reliability. Infrastructure cost is estimable to within a small factor; engineer-days for a migration are not. A ratio that mixes the two inherits the uncertainty of the weaker estimate with none of the visibility.

Reading the frontier as a marginal rate

Once the frontier is small, the useful comparison is between neighbours along it: the slope. Moving from A to B buys 680 ms of p99 latency for some monthly cost and four extra days per report type. Moving from B to C buys another 30 ms for a larger cost increment and a worse operational profile. The second move is where the argument belongs, because it is the point at which the exchange rate between latency and money becomes unattractive.

Stated as marginal rates, the tradeoff is discussable by people who own different concerns: the person who cares about latency can say what the 30 ms is worth, and the person who runs the system can price the operational burden. The frontier does not remove the disagreement, it localises it to two adjacent options and a stated rate.

There is also a frontier shape worth noticing: a large gap between two options in one criterion with everything else equal. That gap is a discontinuity, typically a threshold in the technology — the point where one machine becomes two, where a cache fits in memory, where a query plan changes shape. Discontinuities deserve a prototype rather than a meeting, since the cost of crossing them is what the queueing estimates are for.

Choosing under uncertainty rather than under weights

Weights are a claim about exchange rates that the group usually cannot make honestly. When the uncertainty is in the criteria weights rather than in the option scores, minimax regret gives a decision rule that does not require them.

The regret of option i under criterion weighting w is the difference between the best achievable score under w and the score of option i. Minimax regret selects the option whose worst-case regret across the plausible weighting set is smallest. The calculation is mechanical and the output is a statement of the form “if latency turns out to matter more than the assumed weighting, this option loses at most X relative to the best alternative”.

The rule is conservative, and deliberately so: it protects against the scenario in which the choice turns out badly, at the cost of sometimes forgoing the choice that would have been best. For a decision that is expensive to reverse, that is the right bias. For a reversible decision, the cost of the analysis exceeds the value of the protection, and the earlier depth argument applies.

Limits of the option set

Every method here assumes the option set contains the good answer. In practice the set is assembled from what the participants thought of, and the tradeoff analysis then carefully ranks the second-best ideas.

Two correctives are cheap. Add the option of doing nothing or the minimum, so the frontier has a reference point and the cost of the current design is visible. And derive the options from the sensitivity points identified in the walkthrough rather than from technologies: if the sensitivity point is the concurrency of settlement workers, the options are concurrency settings and workload separation, not “Postgres versus a data warehouse”. The second framing produces options that address the decision; the first produces a technology preference dressed as an evaluation.

A frontier of three options, two of which differ only in a deployment detail, means the real design space was never opened. That is a finding in itself, and usually a more useful one than the ranking.