Conjugate Priors and the Arithmetic of Posterior Updating

Bayes’ rule is a statement about how to slice a joint distribution. For a parameter vector $\theta$ and observed data $D$ it reads $p(\theta \mid D) = p(D \mid \theta),p(\theta) / p(D)$. Only the numerator depends on $\theta$, so the posterior is proportional to the product of the likelihood and the prior, and the work is pushed into the denominator: the evidence $p(D) = \int p(D \mid \theta),p(\theta),d\theta$.

The evidence term is where the cost concentrates

Because a posterior density has to integrate to one, the evidence is whatever the numerator integrates to. That integral runs over the whole parameter space, not over finitely many configurations of the data, and its dimension equals the number of parameters. A single real parameter can be integrated by quadrature to many digits with a few hundred evaluations. Applying the same grid product to ten parameters costs $N^{10}$ evaluations, which at $N = 100$ is $10^{20}$; the approach is unusable well before the parameter count of a realistic model. The denominator is also the marginal likelihood used for model comparison, so it cannot simply be dropped. Everything that follows divides into two families of workaround: represent the posterior parametrically in a family that has a closed form, or compute the integral numerically by sampling, as described in Metropolis-Hastings: Detailed Balance and Proposal Design.

Conjugacy is a closure property, not a coincidence

A prior is conjugate to a likelihood when the posterior lands in the same family as the prior. The pairing is systematic: a Bernoulli or binomial likelihood pairs with a Beta prior, a categorical likelihood with a Dirichlet prior, a Poisson likelihood with a Gamma prior, an exponential likelihood with a Gamma prior, and a normal likelihood with an unknown mean pairs with a normal prior when the variance is treated as known, or with a normal-inverse-gamma prior when both mean and variance are unknown.

The reason the list is finite is that all these families are exponential families. Writing a density as $p(x \mid \eta) = h(x) \exp(\eta \cdot T(x) - A(\eta))$, the log likelihood of $n$ independent observations is linear in the summed sufficient statistic $\sum_i T(x_i)$. If the prior is written over the same natural parameter $\eta$, the product of the two is another member of the family whose natural parameter is shifted by that sum. All the arithmetic of a Bayesian update therefore reduces to adding statistics, and the normalizing constant of the new member supplies the evidence term for free. Conjugacy is a consequence of that additive structure, which is why the closed form survives and why it does not extend to arbitrary likelihoods.

Beta-Binomial: updating as counting

The Beta density over a probability $\theta$ is $p(\theta) \propto \theta^{a-1}(1-\theta)^{b-1}$. For $n$ Bernoulli trials with $k$ successes the likelihood is $\theta^{k}(1-\theta)^{n-k}$, and the product of the two is again Beta:

$$p(\theta \mid k, n) = \mathrm{Beta}(a + k,; b + n - k)$$

The update adds successes to $a$ and failures to $b$, so $a$ and $b$ behave like prior counts. With $a = b = 1$, the uniform prior, the posterior mode is $k/n$, exactly the maximum likelihood estimate; with $a = b = 0.5$ the prior adds half a success and half a failure, and with larger values it pulls harder. A Beta$(1,1)$ prior and a batch of 20 trials containing 12 successes give Beta$(13,9)$, whose mean is $13/22 \approx 0.591$, between the prior mean of 0.5 and the maximum likelihood estimate of 0.6.

The mean obeys a useful identity. It is a weighted average of the prior mean $a/(a+b)$ and the maximum likelihood estimate $k/n$, with weights $a+b$ and $n$:

$$\frac{a+k}{a+b+n} = \frac{a+b}{a+b+n}\cdot\frac{a}{a+b} + \frac{n}{a+b+n}\cdot\frac{k}{n}$$

The ratio $(a+b)/n$ is the exact measure of how much the prior dominates, and it is the number worth quoting when the sensitivity of a posterior to the prior is in question. Small samples mean large prior weight, not because the prior is wrong but because the data have not yet outvoted it.

Sequential updating follows from looking at the posterior parameters: they are sums of counts, so processing the data one observation at a time, in blocks, or in a single batch leaves identical parameters. The cost of a single update is constant in the number of observations, and the state carried between updates is two numbers, which is what makes conjugate models natural for streaming estimation. The predictive probability of a success on the next trial is the posterior mean $(a+k)/(a+b+n)$, which is Laplace’s rule of succession when the prior is uniform.

import math

def log_beta_unnormalized(theta, a, b):
    # log of theta**(a-1) * (1-theta)**(b-1), normalizing constant omitted
    return (a - 1) * math.log(theta) + (b - 1) * math.log(1 - theta)

def update(a, b, x):
    # one Bernoulli observation shifts one count; the posterior stays Beta
    return (a + x, b + 1 - x)

a, b = 1.0, 1.0
for x in [1] * 12 + [0] * 8:   # 20 trials, 12 successes
    a, b = update(a, b, x)

print(a, b)          # 13.0 9.0
print(a / (a + b))   # posterior mean, 0.5909...
print(log_beta_unnormalized(0.6, a, b))   # log posterior at 0.6, up to a constant

Priors that are mixtures of conjugates

A conjugate family contains only distributions of one shape, but a finite mixture of members of the family stretches that. If the prior is a mixture $\sum_j w_j ,\mathrm{Beta}(a_j, b_j)$ and the likelihood is binomial, then the posterior is a mixture of the corresponding Beta posteriors with weights reweighted by the marginal likelihood of each component:

$$w_j’ \propto w_j \int \mathrm{Beta}(\theta; a_j, b_j),\theta^{k}(1-\theta)^{n-k},d\theta$$

The marginal likelihood of a Beta-Bernoulli component has a closed form in terms of the Beta function, so this remains exact and cheap, and the number of components is fixed by the prior rather than growing with the data. The cost is that the representation no longer collapses to a finite set of sufficient statistics: the component weights change with every observation, and the posterior over $\theta$ is not itself a member of the mixture family, since a mixture of Betas is not a Beta.

What closure rules out

Conjugacy ties the shape of the posterior to the shape of the prior. A belief that $\theta$ is probably near 0.2 or near 0.8 and unlikely in between cannot be expressed as a single Beta, because the family has no bimodal members. The usual escape is the mixture just described, which is exact but multiplies the state, or an approximation.

Hierarchical models are where the closed form usually stops. Putting a prior on the hyperparameters $a$ and $b$ of a Beta prior means the posterior over $\theta$ requires integrating over those hyperparameters, and the result is a ratio of functions with no elementary closed form except in special cases. The same failure appears whenever the likelihood is not in an exponential family matching the prior: a logistic regression with a Gaussian prior on the coefficients, or any model whose likelihood involves a nontrivial function of the parameters, has a posterior proportional to a product that cannot be normalized analytically. Population-based and optimization-based substitutes for this integral are covered in Variational Inference and the Evidence Lower Bound.

Improper priors and degenerate edge cases

Two edge cases matter in practice. First, a prior need not be normalizable. The Beta$(0,0)$ prior, sometimes called Haldane’s prior, yields the posterior mean $k/n$ and behaves like a non-informative choice, but it is improper: it assigns infinite mass near 0 and 1. A likelihood with enough data can still produce a proper posterior, yet the evidence term is then undefined, and quantities built on it such as Bayes factors lose their meaning. The Jeffreys prior, proportional to the square root of the determinant of the Fisher information, gives Beta$(1/2, 1/2)$ for the Bernoulli model and is a principled alternative that is proper.

Second, with no observations the posterior is the prior, and the evidence is 1. Any implementation that reports a posterior mean or a marginal likelihood must behave sensibly there: the mean is the prior mean, and the marginal likelihood of an empty data set carries no information at all. The remaining limitation is structural. Conjugacy is preserved only for the pairings that exponential family algebra permits, so it is a technique for the models that admit it rather than a general method for Bayesian computation.