Proper Scoring Rules and Why They Cannot Be Gamed

A scoring rule takes a forecast distribution and the observation that materialised and returns a number. Some scoring rules can be improved in expectation by lying about the forecast; others cannot. The distinction is the entire content of propriety, and it determines whether a model trained to minimise a score is being pushed toward the truth or toward a plausible-looking distortion of it. Getting this wrong is not a subtlety: a model trained under an improper rule converges to forecasts that are systematically overconfident or underconfident, and the training loss looks excellent while the forecast is wrong.

What propriety means

Let $S(P, y)$ be the score for issuing forecast distribution $P$ when $y$ occurs, oriented so that lower is better. The rule is proper if, for every $P$,

$$\mathbb{E}{y \sim P}\left[S(P, y)\right] \le \mathbb{E}{y \sim P}\left[S(Q, y)\right]$$

for every other distribution $Q$; it is strictly proper if the inequality is strict unless $Q = P$. The expectation is taken under the true distribution $P$, so propriety is a statement about the forecast’s own belief: if the forecaster believes $P$, it cannot do better in expectation than to report $P$. A strictly proper rule makes misreporting strictly worse, which means an agent optimising the score has no incentive to shade its forecast.

Propriety is a property of the rule, not of the model. A model trained on a proper rule converges to the best available approximation of the conditional distribution within its hypothesis class, not necessarily the true distribution, but the one closest to it in the divergence the scoring rule induces. Every strictly proper rule induces a Bregman divergence between the true and predicted distributions, and the training problem is the projection of the truth onto the hypothesis class in that divergence. Different proper rules induce different divergences, so they select different members of the class, which is why the choice of score is a modelling decision and not a matter of taste.

Log loss and Brier score as two Bregman divergences

For a binary outcome, the logarithmic score is $-\log p(y)$ and the Brier score is $(p - y)^2$ for $p = P(y=1)$. Both are strictly proper. The log score induces the Kullback–Leibler divergence, and the Brier score induces the squared Euclidean divergence; the difference in the penalties they assign is the difference between an unbounded and a bounded penalty for a confident error.

The log score is unbounded: a forecast of 0.001 for an event that occurs costs about 6.9 nats, and a forecast of $10^{-9}$ costs about 20.7 nats. The penalty grows without limit as the forecast approaches certainty in the wrong direction, which is what enforces sharpness on the right side: a model that is confident and correct gains almost nothing relative to a moderately confident forecast, while a model that is confident and wrong is punished severely. The asymmetry pushes the model’s probability estimates toward the empirical frequencies and away from extremes unless the data really supports them.

The Brier score is bounded above by 1 for a binary outcome and assigns at most 1 to a maximally confident error. It is less sensitive to the far tail, and the corresponding training objective has a gentler gradient when predictions are badly wrong but the loss is already large, so a model trained on Brier loss does not chase rare events as aggressively as one trained on log loss.

The relation between the two for a binary outcome has a clean form. Writing $p$ for the forecast and $y$ for the outcome,

$$-\log p(y) = \text{const} + \frac{(p-y)^2}{2\bar p (1-\bar p)} + O\big((p-y)^3\big)$$

locally, where the denominator involves the values of $p$ near the forecast. The expansion says the two scores agree to second order when the forecast is close to the middle of the range and diverge when it is not: the log score is locally equivalent to a Brier score with a variance that depends on the forecast, which is precisely the heteroscedasticity that makes it unbounded.

import numpy as np

def log_score(p, y, eps=1e-15):
    p = np.clip(p, eps, 1.0 - eps)     # avoid log(0); the clip caps the penalty
    return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))

def brier_score(p, y):
    return np.mean((p - y) ** 2)

The clip in the log score is a real modification, not a numerical nicety. Capping $p$ at $10^{-15}$ bounds the per-observation penalty at roughly 34.5 nats, so a single confident error cannot dominate the mean; without the cap, one observation can produce an infinite training loss and destroy the gradient for the whole batch. The cap moves the rule away from strict propriety near the boundary, which in practice matters less than the arithmetic.

The decomposition and what each part measures

For binary forecasts grouped into $K$ bins with mean forecast $\bar p_k$ and observed frequency $\bar y_k$ over $n_k$ observations, the Brier score decomposes into

$$\underbrace{\sum_k \frac{n_k}{n}(\bar y_k - \bar p_k)^2}{\text{calibration}} - \underbrace{\sum_k \frac{n_k}{n}(\bar y_k - \bar y)^2}{\text{resolution}} + \underbrace{\bar y (1 - \bar y)}_{\text{uncertainty}} .$$

The calibration term is the squared deviation between forecast and observed frequency within bins, so it is a direct measure of miscalibration and is minimised at zero. The resolution term rewards bins whose observed frequency differs from the base rate, so a forecast that separates high-risk from low-risk groups produces a large resolution and a lower score. The uncertainty term depends only on the base rate and cannot be affected by any forecast.

The decomposition is the reason the Brier score is preferred over the log score for diagnostic purposes even though the log score is sharper. The three components have distinct interpretations and can be reported separately, which is what a reliability diagram shows graphically. For the log score no such additive decomposition exists in the probability scale, and the closest analogue is a decomposition of the divergence into a calibration part and a refinement part that is not additive in the same way. The Brier decomposition is also bin-dependent: the split between calibration and resolution changes with the binning, and the total changes too, because binning replaces the per-observation squared errors with their within-bin means. Using many bins reduces the binning artifact but leaves the calibration term noisy; the number of bins has to be balanced against the sample size.

import numpy as np

def brier_decomposition(p, y, bins=10):
    edges = np.quantile(p, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    idx = np.digitize(p, edges[1:-1])
    cal = res = 0.0
    base = y.mean()
    for k in range(bins):
        m = idx == k
        if not m.any():
            continue
        nk = m.sum()                     # equal-count bins here
        cal += nk * (y[m].mean() - p[m].mean()) ** 2
        res += nk * (y[m].mean() - base) ** 2
    n = len(y)
    return cal / n, res / n, base * (1 - base)

Why accuracy and AUC are not scoring rules

A scoring rule takes a distributional forecast. Accuracy takes a hard label derived from the forecast by a threshold, and it is improper for the probability: for a base rate of 0.9, forecasting 1.0 for every observation achieves 90 percent accuracy, while forecasting the true probability of 0.9 for every observation, thresholded at 0.5, also predicts 1.0 and achieves the same accuracy. Every forecast that crosses the threshold scores identically, so accuracy has no gradient in the region where the forecast is most uncertain and it rewards a degenerate forecast in the extreme.

The area under the ROC curve is not a scoring rule at all in the distributional sense: it depends only on the ranking of the scores and is completely insensitive to their calibration. A model that outputs the true probability and one that outputs its square have identical ROC AUC and different log scores, with the second strictly worse. Whatever ROC AUC measures, it does not measure whether the probability is usable as a probability, and the consequences for threshold selection are described in class imbalance and cost-sensitive thresholds.

The practical rule that follows: anything that consumes the probability as a probability (expected-cost decisions, probabilistic forecasts, compound models that use the output as an input) must be fitted and evaluated with a proper scoring rule. Anything that consumes only the ranking can be evaluated with a ranking metric, but should not then be described as producing probabilities. Losses built for a different object illustrate the split: the hinge objective of maximum-margin classifiers is minimised by a boundary rather than by a distribution, so it produces scores whose ordering is meaningful and whose values are not probabilities at all. That is a coherent design, and it is a design that excludes every use which reads the score as a confidence.

Local, strictly proper, and the uniqueness of the log score

Among scoring rules that depend on the forecast only through the probability assigned to the realised outcome, the logarithmic score is the only one that is both strictly proper and local. Locality means the score depends on $p$ only at $y$: the log score is $-\log p(y)$ and nothing else enters. This matters for forecasters who cannot report a full distribution, since a model that must produce a single number for the outcome that occurred can be evaluated on log loss by reading one component, without normalising anything.

The argument for uniqueness rests on the fact that a local proper rule must be a function of the single probability that was assigned, and propriety then forces that function to be the logarithm up to an affine transformation. Any other strictly increasing function of $p(y)$ fails: take $s(p) = 1 - p$ for a binary outcome, which is local and rewards assigning high probability to the realised outcome, but which is maximised by forecasting 1 or 0 regardless of the data, because the expected value $\pi(1-p_y)$ is linear in $p_y$ and therefore maximal at a vertex. Moving from linear to logarithmic is what makes the truth the maximiser rather than the boundary.

The cost of locality is the unbounded penalty, and its practical consequence is that the log score is dominated by the least likely observations. A trained model’s total log loss is often determined almost entirely by a small number of confident errors, and the mean over $n$ observations converges slowly because the distribution of the per-observation loss is heavy-tailed. Brier loss is bounded and has a much better-behaved sample average, at the price of being less discriminating among well-calibrated forecasts that differ only in their tail behaviour. When the tail behaviour is the object of interest, the log score is the sharper instrument; when the tail is noise, it is a source of unstable comparisons between models whose difference is well within the sampling error of the mean.

Bounded rules have a further advantage for model selection: the difference between two models’ Brier scores has a finite variance under mild conditions, so a paired test over observations is straightforward. The same difference computed under log loss can have an infinite variance when the models’ probability estimates approach zero, which makes confidence intervals for the difference unreliable exactly when the models disagree most.

For a multiclass problem the log score is the cross-entropy, and the training problem is the maximum likelihood fit of the same model, which is why the softmax model’s objective in softmax versus one-vs-rest probabilities is the log score in another notation. The equivalence holds only for the log score: training on a multiclass Brier score produces a different fit and a different coefficient vector, and neither is wrong, but the two answer different questions about which member of the hypothesis class is closest to the conditional distribution, in the divergence each score induces.