How Deep to Evaluate: Analysis Cost Against Rework Cost

The evaluation methods worth using cost days of senior time. A full structured review with stakeholders, a prototype, and a capacity study for one decision is five working days of people who are otherwise building the system. Spending that on a decision that can be reversed in an afternoon is waste; not spending it on a decision that fixes a contract for three years is a different kind of waste, paid later with interest. The interesting question is not whether to evaluate but how much, and the answer comes from two properties of the decision rather than from the size of the system.

The inequality that decides the depth

Analysis pays when the expected cost of being wrong, discounted by the chance that the analysis catches it, exceeds the cost of the analysis:

expected loss avoided = P(wrong) x cost_to_undo x share_detectable
analysis is worth its cost when the expected loss avoided exceeds that cost

Every term is a judgement, and two of them are usually underestimated. P(wrong) is underestimated because the design that looks obvious in a meeting has not been tested against the scenarios that matter, and the scenario set is what turns that vague probability into a list of specific ways the design could fail. cost_to_undo is underestimated because it is measured at decision time, when nothing depends on the decision yet.

The third term is the one that makes analysis unnecessary more often than people expect. If no plausible finding would change the choice, the analysis has zero value regardless of how uncertain the situation is. A team arguing between two message broker configurations that both satisfy the throughput scenario, both have the same operational model, and both cost the same has no decision to research; it has a coin to flip. Writing that down as a decision with a revisit trigger ends the discussion faster than a benchmark.

Reversibility is the dominant variable

Decisions differ enormously in what it costs to undo them, and that difference dominates every other consideration. Four classes are worth distinguishing, because they map to different evaluation depths.

Freely reversible decisions live inside one module, one team, and one release. A caching strategy, a serialisation format used only in memory, a retry policy in a client nobody else calls. Undoing these costs hours. Evaluating them at all, beyond reading the code, is usually a mistake.

Costly but reversible decisions cross a boundary without publishing it. Changing an internal HTTP API between two services owned by one team means a coordinated deploy of both, plus a window where both versions run. The undo cost is measured in days and co-ordination, and a review of an hour is proportionate.

Contract decisions are visible outside the team. A published API consumed by customers, a message schema with external subscribers, an on-disk format with existing data, a plugin interface with third-party implementations. Undo cost includes migrating data or supporting two versions indefinitely, and grows with the number of consumers, roughly in proportion to them because each needs its own change and each change has its own co-ordination cost.

One-way doors are structural. Choice of cloud primitives that have no equivalent elsewhere, a data model that encodes a business assumption expected to change, a decision that determines the team topology. These justify the full treatment: scenarios, a walkthrough, a prototype if the uncertainty is numeric, and a written record.

The decision-record practice exists for a reason that is economic rather than archival: a decision whose reasoning is recorded can be revisited on evidence, and a decision whose reasoning is lost gets re-litigated from scratch by whoever notices the consequence first. The record should carry the class, the alternatives rejected, and the trigger that would reopen it.

Four depths, and what each one can answer

The depth ladder is short, and each rung answers a different kind of question.

A solo pass over the scenarios, an hour or two, answers whether the design can satisfy them at all and whether any of them is obviously unaddressed. It costs almost nothing and catches the missing decision — the scenario with no owner, the failure mode nobody designed for.

A peer walkthrough, half a day with two or three engineers, answers where the design’s load-bearing decisions are. It is the utility-tree walkthrough at its cheapest, without stakeholders, and it produces sensitivity points and risks but not a ranking agreed with the business.

A stakeholder evaluation, one to three days with the people who own the requirements, adds the ranking and the argument about measures. Its unique output is a defensible statement of which scenarios are more important, which is the input every later tradeoff needs.

A prototype or measurement answers numeric questions that no discussion settles. Whether a queue can carry the expected burst, whether the chosen database can serve the query shape, whether the latency budget survives three hops. It is the only rung that produces evidence rather than judgement.

The rungs are not cumulative in cost-effectiveness. A prototype that takes three days can replace a two-day stakeholder meeting for a decision that is fundamentally numeric, and vice versa. Choosing the rung is choosing which kind of uncertainty to reduce.

Spikes, and the two ways they fail

A timeboxed prototype is the cheapest source of real numbers about an unbuilt system, because it replaces a guess with a measurement at the cost of a few days. Its rules are strict for a reason.

It answers one question. A spike that asks whether the library works, how fast the query is, and whether the team likes the API produces an answer to none of them, because each answer needs different instrumentation.

It is timeboxed before it starts, with the question stated so the box can close. “Spend two days finding out whether a 200 ms p99 is achievable for this query shape at 10 million rows” has an end condition. “Explore the options” does not.

It is thrown away, or explicitly promoted. The failure mode is the prototype that becomes the product: written under different constraints, without error handling, without tests, and then kept because it works. The second failure is the prototype that answers a question nobody needed answered, which happens when the question was chosen by whoever wanted to write code rather than by the decision.

For performance questions, a prototype does not need production fidelity to be useful if its purpose is to establish an order of magnitude. A 10x margin between the measured cost and the budget is a conclusion; a 1.2x margin is not, and the prototype should say so rather than produce a number that will be quoted as if it were precise.

Stopping rules

Analysis has diminishing returns, and the returns are not smooth: the first hour on a decision usually finds the missing scenario, the second day usually confirms what the first hour found. A stopping rule stated in advance prevents the drift into open-ended evaluation.

Three stopping rules are practical. Stop when the remaining uncertainty cannot change the choice: if every plausible value of the unknown leads to the same design, further measurement is theatre. Stop when the cheapest remaining evidence costs more than the difference between the options: if the two designs differ by a week of work and the experiment to distinguish them costs two weeks, pick either. Stop at the timebox and decide with what is known, recording the uncertainty as an accepted risk with a trigger for revisiting.

The absence of a stopping rule has a cost that is easy to name: decisions not made are also decisions, and they are made later, with less information and less room to move. A schema that nobody finalised for three weeks is a schema that will be finalised during the first week of implementation, under pressure.

Where effort goes when it is misallocated

Three patterns account for most wasted evaluation effort.

Evaluating what is easy to measure. The part of the design with a clean benchmark and a clear number gets studied; the part that is hard to quantify — team boundaries, the operational burden of a new component, the cost of a bespoke protocol — gets decided by whoever speaks last. The corrigible version of this is to give the unmeasurable decisions an explicit treatment in the walkthrough and accept that the treatment is qualitative.

Evaluating too late. An evaluation that happens after the contract is published finds the problem at the moment it becomes expensive. The depth ladder assumes a decision that has not yet been committed; applied afterwards, every rung is a post-mortem.

Evaluating everything at the same depth. A uniform process — every decision gets a two-day review — is easy to administer and wrong in both directions, because it makes cheap decisions expensive and expensive decisions cheap. The classification by reversibility is what makes the process proportionate, and the classification takes minutes.

The economics also run the other way: an architecture with a high rate of irreversible decisions per quarter is a design problem, not an evaluation problem. Systems that keep their contracts narrow and their formats migratable convert one-way doors into ordinary ones, and the evaluation budget falls with them. Reversibility is a property to design for, and the cheapest evaluation is the one the design made unnecessary.