Quality-Attribute Scenarios as the Unit of Architectural Evaluation

A review that opens with “the system must be highly available and easy to maintain” has begun with two claims that cannot be checked, disagreed with, or compared against each other. Availability is a fraction of wall-clock time defined over a window; maintainability is the cost of a class of changes. Neither has a value until someone names the window, the failure, the change, and the budget. The purpose of a quality-attribute scenario is to force those four things onto the table before any design is chosen.

What a scenario has to pin down

Quality attributes are not properties of code. They are properties of a system in an environment under a stimulus. The same deployment is highly available against a disk failure and fragile against a bad configuration push, because the two events differ in detection time, blast radius, and whether the recovery procedure has ever been exercised. A scenario encodes that context so the evaluation compares like with like.

Six fields do the work, and each one closes off a different escape route:

• Source. Who or what generates the stimulus: an end user, a scheduled batch job, a failing NIC, an operator running a migration, a security auditor.
• Stimulus. The event itself, stated concretely: 4,000 concurrent sessions, a 30-second loss of one database replica, a request for a new payment provider, a malformed token.
• Artifact. The part of the system that receives it: the checkout service, the order store, the public API, the whole platform.
• Environment. The conditions under which the stimulus arrives: normal operation, peak load, degraded mode with one zone down, during a rolling deploy.
• Response. The observable behaviour the architecture must produce, phrased without naming internal mechanisms.
• Measure. The number that decides pass or fail.

The measure is where most first drafts collapse. “The response is fast” is not a measure. “The response completes with p99 latency under 300 ms measured at the load balancer, while throughput stays at 2,000 requests per second” is. A measure must name a statistic, a threshold, and the point in the system where it is observed, because latency measured inside the service and latency measured at the client differ by queueing, retries, and connection setup.

Turning drivers into a ranked set

Real systems have a dozen plausible quality attributes and the budget to satisfy two or three of them well. The mapping from business concerns to attributes is one to many: “we lose revenue when the site is slow during promotions” touches availability, performance, and modifiability at once, since the promotion itself is a change. The exercise is to write scenarios until the important ones are covered, then rank them by business consequence rather than by how easy they are to satisfy.

A workable stopping rule is coverage rather than completeness: at least one scenario for each attribute the stakeholders actually named, at least one for each of the six families this method treats — availability, performance, modifiability, security, testability, usability — and at least one that crosses a boundary, since cross-cutting scenarios are where architectural decisions get made. Scenarios that no design could fail are noise; a scenario with no plausible failure mode is a restatement of the status quo.

Ranking matters more than enumeration. Two consequences follow from taking the ranking seriously:

1. The top scenarios constrain the design, and the rest are recorded as context.
2. A design that trades a high-ranked scenario for a low-ranked one is a defect, not a trade-off.

Without a ranking, every discussion of an architecture degenerates into defending whichever attribute the loudest participant cares about. The ranked scenario list is the scoring function that later stages of evaluation depend on, which is why utility trees are built directly on top of it.

A scenario written to be executed

Consider a modifiability scenario for a payment integration. Written loosely it says the system should be easy to extend with new providers. Written as a scenario it becomes a change that a developer could attempt, with a clock attached:

Source:      Product owner requesting a new payment provider
Stimulus:    Add a provider with a redirect flow and webhook callbacks
Artifact:    Checkout service and its persistence layer
Environment: Normal operation, production traffic
Response:    Provider is selectable in configuration; existing providers
             continue to settle in-flight payments
Measure:     Change confined to one module and one migration;
             no edits to order state machine; under 5 engineer-days;
             full regression suite passes unmodified

Two details in that measure carry most of the information. “Change confined to one module” is a structural claim that a dependency analysis can verify, and “regression suite passes unmodified” is a claim about the test suite as a design artifact. A design that requires rewriting the end-to-end tests to accept a new provider fails the scenario even if the feature ships, because the cost of every future provider has just gone up.

The same scenario can be written for a design that does not exist yet. During evaluation, the question is not whether the current code passes but whether the proposed structure would let an unremarkable engineer pass it. That counterfactual is the whole point: scenarios are cheap to write and expensive to retrofit.

Where measures come from without measurement

A scenario needs a number, and in an evaluation there is often nothing to measure. Three sources are legitimate, and they should be labelled so nobody mistakes one for another.

Derived bounds come from the architecture itself. If a request passes through three sequential services, each with a mean latency of 100 ms under nominal load, the composed mean is at least 300 ms before any queueing; that is a floor, not an estimate. Little’s law gives another: concurrency equals arrival rate times residence time, so 500 requests per second at 200 ms residence time means about 100 requests in flight, which sets the connection-pool size. These are arithmetic consequences of the design, and they are the strongest numbers available at evaluation time.

Published limits come from the components. A connection pool bounded at 20, a queue with a fixed depth, a maximum request body of 10 MB, a token lifetime of 15 minutes: each bound is a hard cap on some behaviour, and a scenario’s measure should be consistent with them. A scenario demanding 200 concurrent writes through a pool of 20 will queue; the measure should either account for the queueing or the design should change.

Stakeholder budgets come from the people paying. “A new provider within a week” is a number whether or not it is a good one. It should be recorded with its provenance, since a measure invented by the architecture team and a measure agreed with the product owner carry different weight when the evaluation produces an uncomfortable answer.

What is not legitimate at evaluation time is a throughput figure nobody has measured. Capacity numbers for an unbuilt system are guesses, and a guess in a measure column silently becomes a requirement in the design review.

Failure modes of the format itself

Scenario writing has its own pathologies. The most common is the mechanism leak: a response field that says “the request is routed to the standby replica and the cache is invalidated” has already chosen a design and closed off alternatives, turning an evaluation into a rubber stamp. Responses should stay at the level of observable behaviour, leaving the design free.

The second is the single-scenario system. A list containing only the happy path produces an architecture that is excellent under nominal conditions and undefined everywhere else. Degraded-mode scenarios — one zone down, a dependency slow rather than failed, a certificate expiring mid-deploy — are the ones that expose missing decisions.

The third is measure inflation. Every scenario acquires a p99 target and an availability figure because those are the numbers people know how to write, even for attributes where they are irrelevant. A security scenario’s measure might be “no credential appears in any log line, verified by scanning a full request trace”; a testability scenario’s might be “a new engineer can run the integration suite locally within 30 minutes of cloning”. Numeric, checkable, and not a latency target.

From scenario to decision

A ranked, measurable scenario set is the input that makes every later technique in this cluster meaningful. Dependency analysis tells an evaluator what a change costs but not whether the change matters; the scenario says it matters. Fitness functions are scenarios turned into checks that run on every commit, and they inherit the scenario’s weakness: a check that encodes the wrong measure is worse than no check, because it is believed.

The cost of the technique is a few hours of argument with the people who own the requirements, and the return is that the argument happens once, in the open, instead of repeatedly inside every design discussion. The measure column is the part to defend hardest. It is also the part most often softened, usually by whoever suspects the architecture will not meet it.