Selecting a Decision Threshold from the Confusion Matrix

A binary classifier that emits a probability is a ranking function plus a cut point. The ranking is a property of the model; the cut point is a decision about how to use it, and it belongs to whoever bears the consequences of being wrong. Confusing the two produces the most common evaluation error in applied work: reporting a single accuracy figure for a model whose threshold was chosen on the same data that produced the figure.

The four cells and what each one costs

For a threshold $t$, a row with score $s \ge t$ is called positive. The four cells of the confusion matrix count true positives, false positives, true negatives, and false negatives. Precision is $TP/(TP+FP)$, recall is $TP/(TP+FN)$, and the false positive rate is $FP/(FP+TN)$, which is a property of the negative class alone.

These quantities move in opposite directions as $t$ moves. Raising $t$ admits fewer positives: precision rises because the admitted set is more selective, recall falls because more true positives are left out. The trade is monotone in the sense that both curves are monotone in $t$, but not in the sense that either is a function of the other; precision in particular can be non-monotone in $t$ when the score distribution of the positive class is not unimodal, because raising the threshold can admit a second cluster of positives while dropping a first.

The four cells also differ in how well they are estimated. With a test set of $n$ rows and prevalence $\pi$, the false-negative count is $FN = (1 - \text{recall}) \cdot \pi n$. At $\pi = 0.05$ and $n = 2000$, there are 100 positives, and a recall of 0.9 leaves 10 false negatives; the 95 percent interval on that recall, computed by the Wilson method for a proportion out of 100, spans roughly 0.82 to 0.95. Precision is estimated from the admitted set, whose size depends on the threshold, so precision at an extreme threshold can rest on a handful of rows and carry an interval wide enough to be useless.

Threshold-free summaries and what they hide

The ROC curve plots recall against false positive rate as the threshold sweeps from the top of the score range to the bottom. Its area, AUC, equals the probability that a randomly chosen positive scores above a randomly chosen negative. That interpretation makes AUC independent of the threshold and independent of prevalence, which is why it is popular and why it is misleading when the operating point is far from the middle of the curve.

The independence from prevalence is the specific hazard. A model with an AUC of 0.95 can have a precision near 0.27 at the threshold that achieves 0.9 recall, if the prevalence is 2 percent: of the 2 percent positives, 90 percent are caught, giving 0.018 true positives per row; of the 98 percent negatives, a false positive rate of 5 percent contributes 0.049 false positives per row; precision is $0.018/(0.018+0.049) \approx 0.27$. The same AUC at 50 percent prevalence gives a precision near 0.95. The AUC did not change; the base rate did, and precision is a function of both.

The precision-recall curve carries the prevalence in its values, so its area is prevalence-dependent by construction, which makes it the more honest summary for rare-positive problems and the less comparable one across datasets. The relationship between the two curves is not a monotone transformation of the axes: ROC-AUC and PR-AUC Under Class Imbalance works through the algebra and the conditions under which the two orderings of two models disagree.

A single threshold-free number also hides where the model is useful. Two models with equal AUC can have crossing ROC curves, one better at high recall and the other at high precision; the area integrates over a region that neither deployment will occupy. The curve, or a small table of operating points, is the more informative object, and the scalar is a convenience for ranking candidates before examining the curve.

Choosing the cut from costs, not from the curve

The threshold that minimises expected loss follows from the cost matrix. Let $c_{FP}$ be the cost of a false positive and $c_{FN}$ the cost of a false negative, and let the model output a calibrated probability $p$. Calling positive costs $c_{FP}(1-p)$ in expectation and calling negative costs $c_{FN} p$; the decision is positive when $c_{FP}(1-p) < c_{FN}p$, which rearranges to $p > c_{FP}/(c_{FP}+c_{FN})$.

def cost_optimal_threshold(cost_fp: float, cost_fn: float) -> float:
    """Threshold on calibrated probability that minimises expected cost."""
    return cost_fp / (cost_fp + cost_fn)

With equal costs the rule reduces to $p > 0.5$, which is where the naive 0.5 cut comes from and the only circumstance in which it is correct. With a false negative ten times as expensive as a false positive, the threshold falls to about 0.09: the model should call positive on any row with more than a one-in-eleven chance. That is a large shift from 0.5, and it is invisible in any threshold-free metric.

The derivation depends on calibration. If the model’s scores are not probabilities, the comparison is between a score and a cost ratio with no common scale, and the resulting threshold has no interpretation beyond “the value that minimised cost on the tuning set”. Recalibrating first makes the formula meaningful and makes the threshold transferable to a population with a different base rate, since the cost ratio does not depend on prevalence while the optimal threshold on an uncalibrated score does. Calibrating Scores Before Choosing an Operating Point covers the mapping.

Where costs are unknown, a constrained formulation is often more defensible: pick the lowest threshold that keeps precision at or above a floor, or the highest threshold that keeps recall at or above a floor. These are statements about the operating envelope rather than about a cost model, and they are checkable against the confusion matrix directly.

Reading the curve at the resolution of the decision

A curve plotted over the full score range compresses the region that matters. The horizontal axis of the ROC plot spans the whole false positive rate interval, but a deployment that admits a hundred false positives per million rows occupies the first ten-thousandth of that axis. Points computed at thresholds that admit thousands of false positives are irrelevant to that deployment, and the area under the curve weights them equally with the ones that matter.

The practical correction is to tabulate the confusion matrix at a grid of thresholds and read the table, rather than to inspect the curve. A table with counts at ten thresholds shows the marginal rate of exchange directly: between consecutive rows, the number of additional true positives gained per additional false positive is the local slope, and that ratio is the quantity a cost model prices. Where the ratio falls below the cost ratio, the threshold has moved past the optimum.

The table also exposes the instability that the curve hides. Counts at extreme thresholds rest on small numbers of rows, and the same table computed on a different split of the same data can order the thresholds differently. The slope estimates are therefore noisy exactly where the decision is most consequential, which is the argument for a parametric or smoothed estimate of the trade rather than a direct read of two adjacent rows.

Estimating the threshold without contaminating the estimate

The threshold is a fitted parameter. Selecting it on the same rows used to report performance reproduces the optimism described in Why a Held-Out Test Set Cannot Be Reused, with the added wrinkle that the threshold is chosen to optimise a statistic computed on those rows, so the reported precision and recall at that threshold are biased in the direction of the criterion used to choose it. The bias can be several points when the positive class is small, because the threshold selection is effectively fitting one more parameter to the noise in the tail of the score distribution.

The clean arrangement has three parts. Fit the model on training rows. Choose the threshold on validation rows, either from the cost formula applied to calibrated scores or by scanning the curve against the constraint. Report the resulting confusion matrix on rows that played no part in either step. When data is short, cross-validation supplies the threshold per fold: select inside each training portion, apply to the held-out portion, and average the resulting confusion matrices. That procedure estimates the performance of the whole procedure, threshold included, which is the quantity that matters and the one that nested procedures exist to measure.

A threshold chosen on one population does not transfer to another with a different prevalence. The score distribution of the negative class shifts when the base rate changes, so a fixed cut admits a different fraction of negatives. The cost-optimal probability threshold does transfer, because it is defined on calibrated probabilities and the cost ratio is a property of the decision, not of the population; this is the practical argument for calibrating before fixing a cut point rather than after.