Calibrating Interval Estimates Against Recorded Outcomes

An estimate can be wrong in two independent ways. It can be biased, meaning the central value sits consistently above or below what happens, and it can be miscalibrated, meaning the stated range does not contain the outcome as often as its label claims. Bias is the one people argue about, usually with anecdotes about the last overrun. Calibration is the one that can be measured, because it is a frequency claim and frequencies can be counted.

The distinction matters because the two have different remedies. Bias is corrected by shifting a distribution, which requires knowing the direction and size of past errors. Calibration is corrected by widening or narrowing intervals, which requires knowing how often the stated range actually covered the outcome. Only the second can be established with a record and arithmetic.

What an interval claim commits to

When someone states an eighty percent range of four to six weeks, the claim is not that five weeks is likely. The claim is that across a population of similar statements made under similar conditions, roughly eighty percent of the true values will fall inside the stated bounds and roughly ten percent below and ten percent above.

Under that reading, a single miss proves nothing. A range that fails on one item is entirely consistent with an eighty percent claim, and a range that succeeds on five items in a row also is. Calibration is a property of a set of forecasts, not of any one of them, and it cannot be assessed from the forecast at all. This is why an individual estimate cannot be called overconfident or underconfident in isolation, however confident the discussion sounds.

The two failure directions are not symmetric in their consequences. Overprecision, that is intervals too narrow, causes plans to be treated as commitments and produces the characteristic experience of a schedule that was on track until it was not. Underprecision, intervals so wide as to be useless, is safe but uninformative: a range of two to forty weeks contains the future without helping anyone decide. Since both directions are possible, a measurement that only penalises misses will drive the width to infinity, and a measurement that only rewards narrowness will drive it to zero.

Measuring coverage on a small sample

The measurement needs four recorded fields per item: the interval as stated, the confidence level attached to it, the outcome measured with the same definition, and the date the estimate was made. Without the confidence level, the hit rate is not interpretable, because ninety percent of outcomes inside a range is excellent for a fifty percent claim and poor for a ninety-nine percent claim.

The statistic is the proportion of outcomes that fell inside their stated bounds, compared with the average stated level. Suppose twenty items were estimated with eighty percent ranges and fourteen landed inside. The observed coverage is 0.70. Before concluding that the estimators are overprecise, the sampling noise has to be considered: for a true coverage of 0.8 and twenty independent items, the standard deviation of the observed proportion is sqrt(0.8 · 0.2 / 20), about 0.09. The gap of 0.10 is barely one standard deviation, so the record is consistent with a correct eighty percent claim.

from math import sqrt

hits, n, claimed = 14, 20, 0.80
observed = hits / n
se = sqrt(claimed * (1 - claimed) / n)      # binomial standard error, independence assumed
print(f"observed {observed:.2f}, claimed {claimed:.2f}, se {se:.3f}")
# observed 0.70, claimed 0.80, se 0.089

Roughly speaking, resolving coverage to within five percentage points at eighty percent requires a few hundred items, since the standard error falls as 1/sqrt(n). A single quarter of work will not settle the question. Records of forty to sixty items can distinguish gross overprecision, such as ranges that cover half of what they claim, from reasonable calibration; they cannot distinguish 0.8 from 0.85. Reporting a coverage figure without its sample size invites exactly that over-reading.

Scoring the width alongside the hit

Coverage alone is easy to satisfy by stating bounds so wide that the outcome cannot escape. A proper scoring rule fixes this by charging for the width of the interval and penalising misses in proportion to how far outside the bound the outcome fell. For a central interval at level 1 − alpha:

def interval_score(low, high, y, alpha=0.2):
    # Gneiting-Raftery interval score: width plus alpha/2-weighted miss distance.
    s = high - low
    if y < low:
        s += (2 / alpha) * (low - y)
    elif y > high:
        s += (2 / alpha) * (y - high)
    return s

With alpha = 0.2 for an eighty percent interval, a miss costs ten times its distance beyond the bound. A range from four to six weeks that misses by one week scores 2 + 10 = 12; a range from one to twenty weeks that contains the value scores 19. The score is in the units of the estimate, so it can be averaged over a record and compared between two ways of estimating the same set of items, or between two periods.

Where the forecast is a probability rather than an interval, the Brier score serves the same purpose: the mean squared difference between the stated probability and the outcome coded as one or zero. Both are proper, meaning that the score is optimised in expectation by reporting the honest belief rather than a hedge, which is the property that makes them usable in a setting where the numbers may end up visible to someone with an interest in them.

Reference classes as the correction term

Knowing that intervals are too narrow does not by itself say how to widen them. Reference class forecasting supplies the correction from data rather than judgment. The method takes a population of completed items of the same class, computes for each the ratio of actual to forecast, and treats the resulting distribution of ratios as the error model for a new item of that class. The forecast is then multiplied by the ratio at the desired percentile rather than by the mean.

The reason to use a percentile rather than the mean is that the ratio distribution is right-skewed, so the mean ratio overstates the typical correction while understating the correction needed to reach a high confidence level. A median ratio of 1.2 with a heavy upper tail is compatible with a mean of 1.5, and a plan that needs an eighty percent assurance should be scaled by something near the eightieth percentile of the ratios, not by the average overrun.

Flyvbjerg’s argument for the approach is that it treats optimism as a statistical property of a class rather than as a character flaw to be corrected by exhortation. That framing has a testable implication: if the correction works, coverage should improve without any change in how individual estimates are produced. If coverage does not improve, the reference class was wrong, which is itself information, usually that the population was not homogeneous in the relevant dimension.

Keeping the record clean

Calibration data is fragile in ways that have nothing to do with the estimates. Four contaminations recur.

Re-baselining erases evidence. When a plan slips and the plan of record is restated, the original numbers disappear, and every subsequent measurement compares a forecast against a target that was moved to accommodate it. The record should hold the estimate as first stated, immutable, alongside the delivered date.

Definition drift breaks comparability. An item that was “done” at code complete in one period and “done” at general availability in another cannot be pooled, because the outcome measure changed. The definition belongs in the record.

Scope movement is not estimation error. When the work grows during execution, the eventual duration reflects two effects, and attributing all of it to the estimate makes the correction factor too large and hides the scope process as a source of variation. Recording the scope at estimate time and at delivery separates the two, and the aggregation problem explains why the growth compounds over a long chain.

Selection of what gets recorded matters most. If only large or only visible items are tracked, the population is not the population being forecast, and the correction inherits the difference. A record that includes every item that was forecast, including the ones that were abandoned, is the only one whose coverage statistic means anything.

Once a record exists, the same machinery applies to the promise made from it. A completion date built by sampling past throughput has a coverage claim attached, and that claim can be scored the same way, as described in forecasting completion from throughput.