Week 9: Bayesian Estimation and MAP

Parameters as Random#

Classical (frequentist) statistics treats the unknown parameter $\theta$ as a fixed but unknown constant. An estimator $\hat \theta$ is judged by its behaviour over repeated sampling.

Bayesian statistics treats $\theta$ as a random variable. Before seeing data, you encode your uncertainty about $\theta$ in a prior distribution $p(\theta)$. After observing data $D$, you update to the posterior:

The Bayesian perspective is appealing in ML:

• Uncertainty is explicit: you get a full distribution over parameters, not a point estimate.
• Prior knowledge can be encoded: regularisation in optimisation is often equivalent to a Bayesian prior (e.g., L2 weight decay $\leftrightarrow$ Gaussian prior).
• Sequential updates are natural: today's posterior becomes tomorrow's prior.

The cost: you must specify a prior (where does it come from?), and computing the posterior often requires high-dimensional integration (MCMC, variational inference — beyond this course).

MAP Estimation#

The Maximum A Posteriori (MAPMaximum A Posteriori (estimate)) estimate is the mode of the posterior:

Since $p(D)$ is constant in $\theta$, MAP maximises likelihood $\times$ prior. Compared to MLE:

• MAP = MLE when the prior is uniform (all $\theta$ equally plausible a priori).
• MAP shrinks toward the prior when data are scarce; the influence of the prior diminishes as $n$ grows.
• MAP is a point estimate — it gives you one $\theta$, not a full distribution. The posterior may be multimodal (multiple plausible parameter values); MAP picks one mode and ignores the rest.

MAP is widely used in practice because it adds a regularisation interpretation while retaining the optimisation framework (just add $\log p(\theta)$ to the objective).

Beta–Binomial#

The classic conjugate pair for a probability parameter. Suppose we model a coin's bias $p \in [0, 1]$:

• Prior: $p \sim \text{Beta}(\alpha, \beta)$ with density $f(p) \propto p^{\alpha-1} (1-p)^{\beta-1}$ for $p \in [0, 1]$.
• Likelihood: Observe $k$ heads in $n$ flips: $L(p) \propto p^k (1-p)^{n-k}$.
• Posterior: $p \mid \text{data} \sim \text{Beta}(\alpha + k,\; \beta + n - k)$.

The Beta prior is conjugate to the Bernoulli/Binomial likelihood — the posterior is also Beta. The hyperparameters act as pseudo-counts: $\alpha - 1$ is like having seen $\alpha - 1$ imaginary heads, and $\beta - 1$ imaginary tails, before the experiment.

Worked example. Prior: $\text{Beta}(2, 2)$ (mild belief that the coin is roughly fair — equivalent to having seen 1 head and 1 tail). Observe $k = 7$ heads in $n = 10$ flips.

Posterior: $\text{Beta}(2 + 7,\; 2 + 3) = \text{Beta}(9, 5)$.

The posterior mean is $\frac{\alpha + k}{\alpha + \beta + n} = \frac{9}{14} \approx 0.643$, and the MAP is $\frac{\alpha + k - 1}{\alpha + \beta + n - 2} = \frac{8}{12} \approx 0.667$.

Compare to MLE: $\hat p_{\text{MLE}} = 7/10 = 0.7$. The Beta$(2,2)$ prior pulls the estimate slightly toward 0.5. With $n = 10$ the prior is still visible; with $n = 1000$ heads and $500$ tails, the prior would be negligible.

Beta distribution shapes:

• $\text{Beta}(1, 1)$ = Uniform$(0, 1)$ — no prior information.
• $\text{Beta}(0.5, 0.5)$: U-shaped, favours extremes (Jeffreys prior).
• $\text{Beta}(\alpha > 1, \beta > 1)$: unimodal, centred near $\alpha/(\alpha+\beta)$.

Normal Mean with Known Variance#

The second classic conjugate pair. Assume $X_i \sim \mathcal{N}(\mu, \sigma^2)$ with $\sigma^2$ known.

• Prior: $\mu \sim \mathcal{N}(\mu_0, \tau_0^2)$.
• Likelihood: $\bar x \sim \mathcal{N}(\mu, \sigma^2 / n)$ (sufficient statistic).
• Posterior: $\mu \mid \text{data} \sim \mathcal{N}(\mu_n, \tau_n^2)$, where

The posterior mean is a precision-weighted average of the prior mean and the sample mean. Precisions ($1/\text{variance}$) add: posterior precision = prior precision + data precision.

• If the prior is very tight ($\tau_0^2$ small), the posterior mean stays near $\mu_0$ unless the data are overwhelming.
• If the prior is diffuse ($\tau_0^2$ large), the posterior mean approaches $\bar x$ and the posterior variance approaches $\sigma^2 / n$.

Posterior Predictive (lite)#

The posterior predictive distribution for a new observation $\tilde x$ averages the sampling distribution over the posterior:

This is not $p(\tilde x \mid \hat \theta)$ — plugging in a point estimate ignores parameter uncertainty. For the Beta–Binomial, the posterior predictive probability of heads on the next flip is the posterior mean:

This is a natural "smoothed" estimate that accounts for both the data and prior uncertainty.

Knowledge check#

Browser lab#

Beta–Binomial posterior density for different priors given the same dataset.

import numpy as np
import matplotlib.pyplot as plt
from math import gamma as math_gamma

def beta_pdf(x, a, b):
    """Beta(a, b) density at points x (scalar or array)."""
    x = np.asarray(x, dtype=float)
    log_pdf = (a - 1) * np.log(x) + (b - 1) * np.log(1 - x)
    log_pdf -= np.log(math_gamma(a)) + np.log(math_gamma(b)) - np.log(math_gamma(a + b))
    valid = (x > 0) & (x < 1)
    result = np.zeros_like(x)
    result[valid] = np.exp(log_pdf[valid])
    return result

n, k = 10, 7

prior_configs = [
    ("Beta(1,1) — Uniform", 1, 1, "tab:blue"),
    ("Beta(2,2) — mild fairness", 2, 2, "tab:orange"),
    ("Beta(5,2) — favour heads", 5, 2, "tab:green"),
    ("Beta(0.5,0.5) — Jeffreys", 0.5, 0.5, "tab:red"),
]

p_grid = np.linspace(0.001, 0.999, 300)

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

for label, a, b, color in prior_configs:
    prior_pdf = beta_pdf(p_grid, a, b)
    axes[0].plot(p_grid, prior_pdf, color=color, linewidth=2, label=label)

axes[0].set_title("Priors")
axes[0].set_xlabel("p"); axes[0].set_ylabel("Density")
axes[0].legend(fontsize=8)

for label, a, b, color in prior_configs:
    post_a, post_b = a + k, b + (n - k)
    post_pdf = beta_pdf(p_grid, post_a, post_b)
    axes[1].plot(p_grid, post_pdf, color=color, linewidth=2, label=label)
    mle = k / n
    map_est = (post_a - 1) / (post_a + post_b - 2) if post_a > 1 and post_b > 1 else np.nan
    axes[1].axvline(mle, color="black", linestyle=":", alpha=0.5)
    if not np.isnan(map_est):
        axes[1].scatter([map_est], [beta_pdf(map_est, post_a, post_b)],
                        color=color, s=50, zorder=5, marker="D")

axes[1].axvline(mle, color="black", linestyle=":", alpha=0.5, label=f"MLE={mle:.2f}")
axes[1].set_title(f"Posteriors — {k} heads / {n} flips")
axes[1].set_xlabel("p"); axes[1].set_ylabel("Density")
axes[1].legend(fontsize=8)

plt.tight_layout()
plt.show()

for label, a, b, _ in prior_configs:
    post_a, post_b = a + k, b + n - k
    post_mean = post_a / (post_a + post_b)
    print(f"{label:30s} posterior: Beta({post_a},{post_b})  mean={post_mean:.3f}")