Week 5: Bayes' Rule and Generative Classification

Bayes' Rule#

Bayes' rule tells us how to reverse a conditional probability — to go from $P(\text{data} \mid \text{hypothesis})$ to $P(\text{hypothesis} \mid \text{data})$:

The denominator is the total probability of the data: $P(D) = \sum_{\theta'} P(D \mid \theta') P(\theta')$ for discrete hypotheses. In the continuous case the sum becomes an integral, but the structure is identical.

The four ingredients have standard names:

| Symbol | Name | Meaning | |---|---|---| | $P(\theta)$ | Prior | Belief about $\theta$ before seeing data | | $P(D \mid \theta)$ | Likelihood | How probable the observed data is, assuming $\theta$ is true | | $P(\theta \mid D)$ | Posterior | Updated belief about $\theta$ after seeing data | | $P(D)$ | Evidence (marginal likelihood) | Overall probability of the data, averaging over all possible $\theta$ |

Bayes' rule is the optimal way to update beliefs when you receive new information. It follows directly from the definition of conditional probability:

The proportional form $P(\theta \mid D) \propto P(D \mid \theta) \, P(\theta)$ is all you need when comparing hypotheses — the denominator $P(D)$ is the same normalising constant for every $\theta$.

Prior, Likelihood, Posterior#

Discipline with vocabulary. Bayes' rule is simple algebraically, but decades of confusion stem from sloppy naming. Be precise:

• Prior = your model of the world before this experiment. It can be uniform (maximal uncertainty), informed by previous experiments, or subjective.
• Likelihood = the forward model — if $\theta$ were true, how often would we see data like $D$?
• Posterior = the prior re-weighted by the likelihood. When the likelihood strongly favours a particular $\theta$, the posterior concentrates there, even if the prior was diffuse.

Worked example — coin bias estimation. You have a coin that may be fair ($\theta = 0.5$) with prior $P(\theta = 0.5) = 0.7$ or biased toward heads ($\theta = 0.8$) with prior $P(\theta = 0.8) = 0.3$. You flip it three times and observe H, H, T (two heads, one tail).

Likelihood of this sequence under each hypothesis:

Evidence: $P(D) = 0.125 \cdot 0.7 + 0.128 \cdot 0.3 = 0.0875 + 0.0384 = 0.1259$.

Posterior:

The data slightly favours $\theta = 0.8$, but the prior was strong for $\theta = 0.5$, so the posterior remains tilted toward the fair coin. With more flips, the likelihood eventually dominates.

Worked Medical Test — Base-Rate Neglect#

A disease affects 1 in 1000 people (prior = 0.001). A test is 99% accurate: it returns positive for 99% of diseased people and negative for 99% of healthy people.

A patient tests positive. What is the probability they actually have the disease?

Confusion arises because people substitute $P(\text{positive} \mid \text{disease}) = 0.99$ for $P(\text{disease} \mid \text{positive})$, ignoring the base rate. Let us apply Bayes:

• Prior: $P(D) = 0.001$.
• Likelihood: $P(+ \mid D) = 0.99$ (true positive rate).
• False positive: $P(+ \mid \neg D) = 0.01$ (1% of healthy people test positive).
• Evidence: $P(+) = P(+ \mid D)P(D) + P(+ \mid \neg D)P(\neg D) = 0.99 \cdot 0.001 + 0.01 \cdot 0.999 = 0.00099 + 0.00999 = 0.01098$.

Now Bayes:

Despite a "99% accurate" test, a positive result implies only about a 9% chance of actually having the disease. Why? Because the disease is so rare that even the small false-positive rate (1%) swamps the true positives. This is base-rate neglect — a pervasive reasoning error in medicine, law, and data science. Always ask: how rare is the thing I am testing for?

Generative Classification Sketch#

A generative classifier models the joint distribution $P(x, y)$ and then uses Bayes' rule to compute $P(y \mid x)$ for prediction:

Since $P(x)$ does not depend on $y$, the classifier that maximises the posterior is:

This is the MAPMaximum A Posteriori (estimate) (Maximum A Posteriori) decision rule — pick the class $y$ that makes the observed features $x$ most probable, weighted by how common each class is a priori.

Worked example — tiny classifier. Suppose we have two classes and one binary feature:

| | Class A ($y = 0$) | Class B ($y = 1$) | |---|---|---| | $P(y)$ | 0.6 | 0.4 | | $P(x=0 \mid y)$ | 0.9 | 0.3 | | $P(x=1 \mid y)$ | 0.1 | 0.7 |

Given $x = 1$, which class does MAP predict?

• $P(x=1 \mid y=0) \cdot P(y=0) = 0.1 \cdot 0.6 = 0.06$
• $P(x=1 \mid y=1) \cdot P(y=1) = 0.7 \cdot 0.4 = 0.28$

$\hat y = 1$ (Class B). Even though Class A is more common (prior 0.6), the likelihood strongly favours B when $x = 1$, so the posterior shifts.

Given $x = 0$:

• Class A: $0.9 \cdot 0.6 = 0.54$
• Class B: $0.3 \cdot 0.4 = 0.12$

$\hat y = 0$. Now the prior and likelihood both favour A.

Naive Bayes Mention#

The generative classification framework requires $P(x \mid y)$ — the probability of the entire feature vector given the class. For high-dimensional $x$, estimating this joint directly is infeasible. Naive Bayes makes the conditional independence assumption from Week 4:

Each feature is modelled independently given the class. This is almost always false, yet it works well: even with wrong probability estimates, the ranking of classes often remains correct. Naive Bayes is the simplest generative classifier and is widely used for text classification (spam detection, sentiment analysis) as a fast, interpretable baseline.

Knowledge check#

Browser lab#

Bar charts for a discrete Bayesian update: prior → unnormalised posterior → normalised posterior for three hypotheses.

import numpy as np
import matplotlib.pyplot as plt

hypotheses = ["θ=0.3", "θ=0.5", "θ=0.7"]
prior = np.array([0.3, 0.5, 0.2])
theta_vals = np.array([0.3, 0.5, 0.7])

# Observe 7 heads in 10 flips
n, k = 10, 7
likelihood = theta_vals**k * (1 - theta_vals)**(n - k)

unnorm_posterior = prior * likelihood
evidence = unnorm_posterior.sum()
posterior = unnorm_posterior / evidence

fig, axes = plt.subplots(1, 3, figsize=(12, 4))
colors = ["tab:blue", "tab:orange", "tab:green"]
for ax, vals, title in zip(axes,
    [prior, unnorm_posterior, posterior],
    ["Prior", f"Prior × Likelihood\n(unnormalised)", "Posterior\n(normalised)"]):
    ax.bar(hypotheses, vals, color=colors, alpha=0.8)
    for i, v in enumerate(vals):
        ax.text(i, v + max(vals)*0.02, f"{v:.3f}", ha="center", fontsize=9)
    ax.set_ylim(0, max(vals) * 1.15)
    ax.set_title(title)

fig.suptitle(f"Bayesian Update — {k} heads in {n} flips", fontsize=14)
plt.tight_layout()
plt.show()

print(f"Evidence P(D) = {evidence:.4f}")
print("Posterior:", dict(zip(hypotheses, np.round(posterior, 4))))
print(f"MAP estimate: θ = {theta_vals[np.argmax(posterior)]}")