Class Imbalance, Class Priors, and Cost-Sensitive Thresholds

A classifier that outputs a probability and a classifier that outputs a decision are different objects, and imbalance problems are where the difference bites. The probability is a statement about the joint distribution; the decision is the argmin of an expected loss that includes the relative cost of the two error types. Confusing the two produces a family of familiar mistakes: resampling the training set to equalise the classes changes the probabilities the model estimates, thresholding at one half ignores the prior, and evaluating with accuracy rewards a constant prediction.

The decision rule already contains the prior

For a two-class problem with costs $C_{10}$ for a false positive and $C_{01}$ for a false negative, the expected cost of predicting class 1 is $(1-\pi) C_{10}$ and of predicting class 0 is $\pi C_{01}$, where $\pi$ is the posterior probability of class 1. Predicting class 1 pays off when

$$\pi C_{01} > (1-\pi) C_{10}, \qquad \text{that is,} \qquad \pi > \frac{C_{10}}{C_{10} + C_{01}} .$$

The threshold is a ratio of costs, not one half, and it is independent of the class prior, the prior already lives inside $\pi$. When a false negative is ten times as expensive as a false positive, the threshold is $1/11 \approx 0.091$. At a base rate of one percent, a well-calibrated model then flags a large share of the population, and that is the correct decision under the stated costs rather than a sign of a broken model.

The consequence is that “optimising the threshold” is not a post-processing hack that compensates for imbalance. It is the decision step of a decision-theoretic procedure, and it requires a calibrated probability. If the posterior is miscalibrated, the threshold computed from costs is applied to the wrong quantity and neither the threshold nor the calibration is doing what it appears to do.

Resampling changes the estimand

Oversampling the minority class, by duplication or by interpolation between neighbours, as in the SMOTE construction, or undersampling the majority class changes the empirical class frequencies in the training set. Since the model maximises likelihood on whatever data it sees, the fitted intercept converges to the log-odds implied by the resampled frequencies, not the population ones. In the logistic case this is exact and simple: the intercept moves by the log ratio of the sampling rates. If the population has 1 percent positives and the resampled set has 50 percent, the intercept shifts by $\log(0.5/0.01) \approx 3.9$ in favour of the positive class, and every predicted probability is inflated accordingly.

The slope coefficients are not affected by class-balanced sampling when the model is correctly specified, because the conditional distribution of the features given the class is unchanged by sampling on the class. That asymmetry, with slopes preserved and the intercept shifted, is what makes the correction possible: subtracting the log ratio of sampling rates from the intercept recovers the population probabilities. The correction is exact only under the assumed functional form; a boosted tree model fitted to resampled data is not a logistic model and its score transformation is not a single additive constant, though the same monotone distortion in the scores appears.

Duplicating minority rows rather than generating synthetic ones also changes the effective sample size without adding information. The likelihood terms for duplicated rows are simply squared, which makes the fit more confident about the minority region than the data warrants, and any standard error computed from the resampled set is wrong because the rows are not independent.

import numpy as np

def prior_correction(pi_sampled, rate_pop, rate_sample):
    """Map probabilities fitted on a resampled set back to population scale."""
    # logit_adj = logit(pi_sampled) - log(rate_sample / rate_pop)
    pi = np.clip(pi_sampled, 1e-12, 1 - 1e-12)
    logit = np.log(pi / (1 - pi))
    logit -= np.log(rate_sample / rate_pop)
    return 1.0 / (1.0 + np.exp(-logit))

The correction is a monotone transformation, so it cannot change the ranking of the scores. Every metric that depends only on ordering, such as the area under the ROC curve or the ranking loss, is invariant to it, which is why resampling looks harmless when evaluated by ROC AUC and harmful when evaluated by anything that reads the probabilities.

Why ROC AUC hides the problem

The ROC curve plots the true positive rate against the false positive rate across every possible threshold. Both rates are conditional on the class, so they are computed within each class separately and the class prior cancels out. The area under that curve is therefore a summary of separability that does not depend on prevalence: the same model applied to a population with a one percent base rate and one with a fifty percent base rate has the same ROC AUC, and consequently the same usefulness as a ranking device, while the precision at any fixed threshold differs by a large factor.

Precision-recall curves are sensitive to prevalence because precision depends on the number of false positives, which grows with the size of the negative class. A model with an ROC AUC of 0.95 can have a precision at 50 percent recall of 0.16 at a one percent base rate, purely from the arithmetic of the confusion matrix, and this is not model failure, it is the base rate. Reporting ROC AUC alone on an imbalanced problem answers a question about ranking that nobody asked. The mean predicted probability is the quantity the ranking metric is blind to, and it drifts with the base rate even when the ranking is unchanged.

Calibration is the property that decision rules rely on

A model is calibrated when, among the observations assigned a probability of 0.2, the event occurs about 20 percent of the time. Calibration is a statement about the marginal relationship between the score and the outcome, and it is separate from discrimination: a model can rank perfectly and be badly calibrated if its scores are a monotone but distorted function of the true probability.

The reason calibration matters here is that the threshold derived from costs is a specific number. A misordering of scores does not change which observations are above 0.091 in a way that matters, but a model whose scores cluster near 0.9 for everything makes every threshold useless. Reliability diagrams bin the predictions and plot the observed frequency against the mean predicted value; isotonic regression fits a monotone piecewise-constant map from scores to frequencies, and Platt scaling fits a two-parameter logistic map. Both are fitted on held-out data, and both change the scores in a way that would invalidate any metric computed on the same data used to fit the calibrator.

The reason the intercept carries the prior at all is visible in the likelihood: the class balance enters a logistic model only through a constant added to the linear predictor, which is why logistic regression as conditional likelihood writes the intercept as the log prior odds when no covariate is informative.

For logistic regression fitted by maximum likelihood on the population distribution, calibration holds by construction, up to estimation error. For support vector machines without probability output, for boosted trees, and for neural networks, calibration is empirical rather than structural, and the deviation tends to be systematic: modern deep models are frequently overconfident, with the overconfidence concentrated in the high-confidence bins. The threshold derived from a cost ratio applied to such scores is not the optimal decision rule for the stated costs, and the error direction is toward over-predicting the majority class in some bins and the minority class in others depending on where the miscalibration lies.

Working with the classes rather than against them

A separate approach changes the loss to reflect the costs directly, weighting the positive example’s contribution to the objective by $C_{01}/C_{10}$. In the logistic case the weighted negative log-likelihood is

$$\sum_i w_{y_i} \left[\log(1 + e^{x_i^\top \beta}) - y_i x_i^\top \beta\right],$$

which is the same convex objective with a per-observation multiplier. The weighted objective is the correct likelihood for a population where the class-conditional densities are unchanged but the prior is the one implied by the weights, so the same intercept-shift story applies. The difference is that the weighting happens during fitting rather than afterwards, and the resulting model can devote its capacity to the minority region instead of being dominated by the majority.

A hard threshold is the simplest decision rule but rarely the right one when the costs are uncertain. If the cost ratio is known only within a factor of two, the threshold is known only within a factor of two, and a decision that flips on the exact number is fragile. The expected-cost curve as a function of threshold is usually flat near its minimum, so a wide range of thresholds is nearly optimal, which is a reason to pick a threshold in the middle of the flat region rather than to spend effort refining the cost estimate.

The choice of the quantity being optimised follows the same logic. A threshold tuned on log loss and a threshold tuned on squared probability error differ, because the two losses weight errors near the base rate differently, and only one of them is a proper scoring rule in the sense of rewarding an honest probability, the subject of proper scoring rules. Deciding which to use requires deciding whether the downstream consumer reads the probability or only the label.

Ranking a candidate set for review is a different problem from classifying all of it, and the size of the review budget is usually the binding constraint. Under a fixed budget of the top $k$ scores, the decision is a top-$k$ selection and the relevant question is precision within that set, which is again a function of the base rate. The two framings, one a fixed threshold derived from costs and the other a fixed budget derived from capacity, produce different selected sets, and conflating them is how a threshold tuned for one deployment gets applied to another.