ROC-AUC and PR-AUC Under Class Imbalance

Two curves summarise the same ranking and disagree about which model is better. The disagreement is not a bug in either; the two areas integrate over different axes, and one of those axes changes when the class balance changes while the other does not. Understanding which axis is which resolves most of the confusion about which metric to report on a dataset with a rare positive class.

The axes and the invariance

The ROC curve plots true positive rate against false positive rate as the threshold sweeps. Both axes are conditional on the true class: TPR is computed over positives, FPR over negatives. Neither denominator contains the prevalence, so neither coordinate changes when the class balance changes while the score distributions within each class stay the same. Consequently the whole curve, and its area, are invariant to prevalence.

The precision-recall curve plots precision against recall. Recall is conditional on positives, but precision is $TP/(TP+FP)$, whose denominator is the set of predicted positives, which mixes both classes. Precision therefore depends on prevalence directly: at a fixed TPR and FPR, precision is $\text{TPR} \cdot \pi / (\text{TPR} \cdot \pi + \text{FPR} \cdot (1-\pi))$, where $\pi$ is the prevalence. Halving $\pi$ roughly halves the numerator and barely touches the denominator, so precision falls.

The AUC of the ROC curve has the clean interpretation: the probability that a random positive is ranked above a random negative. This is a pairwise quantity over one positive and one negative, so prevalence never enters. The AUC of the PR curve has no equally clean interpretation; it is a weighted average of precisions over the recall range, and its value depends on the interpolation used between operating points, which is a known source of disagreement between implementations.

Why ROC-AUC looks fine on a problem that is not fine

At 1 percent prevalence, a model with a false positive rate of 1 percent produces one false positive per 100 negatives, which is about 0.0099 false positives per row, against 0.009 true positives per row from a recall of 0.9. Precision at that point is about 0.009/(0.009+0.0099), near 0.48. The ROC coordinates for that point are (0.01, 0.9), which look excellent on a plot whose axes both span 0 to 1: the point is near the top-left corner. The plot is correct and the visual impression is misleading, because the region near FPR = 0 is compressed into a sliver of the axis while it contains almost all the rows the deployment will actually see.

The compression is the mechanism. FPR ranges over the same interval regardless of prevalence, but the number of false positives produced by a given FPR scales with the negative count, which is nearly all the rows when positives are rare. At 1 percent prevalence, moving along the ROC axis from FPR 0 to FPR 0.5 adds half a false positive per row while the true positive count cannot exceed 0.01 per row; every operating point past the first sliver of the axis predicts positive for a large share of the population. The ROC area weights the axis uniformly, so it spends most of its weight on regions where the classifier is calling a large fraction of all rows positive.

The PR curve’s x-axis is recall, which is also prevalence-independent, but its y-axis carries the prevalence penalty, so the curve sits low on the plot when positives are rare. A model with ROC-AUC 0.95 can have PR-AUC 0.3 on the same data. Neither number is wrong; the PR number is the one that reflects the experience of using the model.

When the two orderings disagree

Two models with crossing curves can have equal areas and different shapes, and the area comparison then depends on which axis is integrated. Davis and Goadrich showed that a curve dominating in ROC space dominates in PR space, so the orderings agree when one curve dominates throughout. Disagreement arises exactly when the curves cross, which is the common case for models of different capacity: a high-capacity model ranks the top of the positive distribution well and produces scattered scores in the bulk, while a low-capacity model is worse at the top and more consistent overall.

In that situation the ROC area can favour the high-capacity model, because its advantage at low FPR is worth as much area as the other model’s advantage elsewhere, while the PR area favours whichever model does better in the high-precision region, since that region occupies a larger share of the PR axis. The practical rule is that the metric must be chosen from the operating region: if the deployment can tolerate only a small number of false positives per thousand rows, the comparison should be restricted to that region rather than integrated over the whole curve.

Restricting the region is a partial area computation, and partial areas have their own normalisation question. A partial ROC area over FPR in [0, 0.05] can be reported as raw area or normalised by the width of the interval; the raw value is not comparable across different interval widths, and the normalised value can exceed a full-curve comparison’s meaning. Naming the interval alongside the number is what makes it interpretable.

The baseline problem

A random classifier has ROC-AUC 0.5 regardless of prevalence, which makes the ROC area easy to read as a fraction of achievable skill. Its PR-AUC is not 0.5; it equals the prevalence. At 1 percent prevalence, a random ranker has PR-AUC 0.01, and a model with PR-AUC 0.3 is thirty times better than chance. The baselines differ, so the two numbers are not on a common scale and cannot be compared across datasets with different prevalence.

The PR baseline’s dependence on prevalence is also why PR-AUC is not comparable across datasets: a PR-AUC of 0.5 on a 10 percent prevalence problem and 0.5 on a 1 percent problem represent different amounts of skill. The ROC area does not have this problem, which is the argument for using it when comparing across populations and against it when the population is fixed and the operating region is not the whole curve.

import numpy as np

def pr_auc(y_true, score):
    """Average precision: sum over thresholds of precision weighted by recall change."""
    order = np.argsort(-score)
    y = np.asarray(y_true)[order]
    if y.sum() == 0:
        return 0.0
    tp = np.cumsum(y)
    fp = np.cumsum(1 - y)
    precision = tp / (tp + fp)
    recall = tp / tp[-1]
    # weight each precision by the recall gained at that step
    return float(np.sum(np.diff(np.concatenate(([0.0], recall))) * precision))

The step-weighted sum is average precision, which equals the area under the step-wise PR curve and differs from trapezoidal interpolation of the same points. Trapezoidal interpolation on a PR curve overstates the area when the curve is convex, which it usually is, so the two conventions give different numbers for the same model. Average precision is the more defensible of the two because it does not assume linearity between operating points.

Recomputing the curve when the prevalence is known

Precision at a fixed operating point is recoverable from the ROC coordinates and the prevalence, so a PR curve can be reconstructed for a population whose base rate differs from the one measured. The relation is exact algebra, not approximation, and it requires no new data: take each ROC point, convert with the known prevalence, and plot the resulting precision against recall.

The reconstruction is useful when the test set’s prevalence does not match the deployment’s, which happens when the test set was enriched to obtain enough positives. An enriched set of 10 percent positives used to measure a model intended for a 1 percent population produces a PR curve at 10 percent; recomputing at 1 percent gives the curve the deployment will see, and it is uniformly lower. The ROC curve is unaffected, which is a second illustration of the invariance.

The catch is that the model itself may change under the new prevalence. If the model was trained on the enriched set, its ranking was optimised for that distribution, and the reconstruction assumes the ranking is unchanged. That assumption holds only if the model’s scores were not fitted to the enriched prevalence, which is generally false. Reconstruction is therefore valid for reweighting an evaluation and not for transferring a model.

Reporting both, and what to do with the disagreement

Reporting both areas with their baselines is the minimum that makes a comparison interpretable. A model described as “AUC 0.94” on a 2 percent prevalence problem has told the reader almost nothing about whether it is usable; the same model described as “ROC-AUC 0.94, PR-AUC 0.31 against a 0.02 baseline” has told the reader that the ranking is strong and the precision is modest, which is the honest summary.

When the two disagree about which of two models is better, the disagreement is a signal to look at the operating region rather than to pick the metric that gives the preferred answer. The confusion matrix at the deployment threshold, covered in Selecting a Decision Threshold from the Confusion Matrix, resolves the question directly: whichever model has the better precision at the recall the deployment needs is the better model for that deployment, regardless of either area.

The areas also say nothing about calibration. A model can have a high ROC area and badly calibrated probabilities, because the area depends only on the ordering and the ordering is unchanged by any monotone transformation of the scores. Two models with identical curves can differ in whether a score of 0.7 means a 70 percent chance, and that difference matters as soon as the threshold is chosen by cost rather than by position on the curve. Calibrating Scores Before Choosing an Operating Point covers the diagnostic and the correction.