Week 14: Capstone — Probabilistic Language of ML

Two Target Dialects#

This course was designed to teach the probabilistic language that two applied courses depend on. By now, you can read the math in both:

Reinforcement Learning speaks in expectations over stochastic trajectories: $\mathbb{E}_\pi[\sum_{t} \gamma^t R_t]$, value functions $V(s) = \mathbb{E}[G_t \mid S_t = s]$, and conditional transition probabilities $P(s' \mid s, a)$. This is Weeks 3–4 and 7 in action.

Generative Models speak in MLE, KL divergence, and latent-variable bounds: $\log p_\theta(x) \ge \mathbb{E}_q[\log p_\theta(x \mid z)] - D_{\mathrm{KL}}(q \parallel p(z))$. This is Weeks 8–9 and 11 in action.

The rest of this capstone maps the specific connections and gives you case prompts to practise choosing the right tool.

RL Lens#

Reinforcement learning is built on probability at every level. Here is the mapping:

• Stochastic transitions: $P(s' \mid s, a)$ is exactly a conditional probability — Week 4 (conditioning). The Markov property $P(s_{t+1} \mid s_t, a_t, s_{t-1}, \ldots) = P(s_{t+1} \mid s_t, a_t)$ is a conditional independence statement (Week 13).
• Expected returns: The value function $V^\pi(s) = \mathbb{E}_\pi[\sum_{t=0}^\infty \gamma^t R_t \mid S_0 = s]$ uses linearity of expectation (Week 3) to decompose into immediate reward plus discounted future value (the Bellman equation). No independence is assumed — linearity handles the stochastic dependencies.
• Policy as a distribution: $\pi(a \mid s)$ is a conditional probability distribution over actions. Exploration strategies (epsilon-greedy, entropy bonuses) are about increasing the variance/entropy of this distribution (Week 11).
• Sampling and Monte Carlo: Experience replay and rollout-based methods estimate expectations via sample averages — the LLN (Week 7) justifies this.
• Uncertainty in model-based RL: Learned dynamics models $\hat P(s' \mid s, a)$ can be Bayesian (Week 9), with posterior uncertainty propagated through planning — or bootstrapped (Week 10).

The RL course (course index) starts from exactly this probabilistic language. You are now equipped to read the Bellman equation not as a mysterious recursion but as an application of expectation, conditioning, and linearity.

Generative Models Lens#

Generative models aim to learn $p_\theta(x)$, the distribution of data. The probabilistic foundations map as follows:

• Maximum likelihood (Week 8): Training most generative models maximises $\sum_i \log p_\theta(x_i)$. This is MLE. The connection to KL (Week 11) tells us this minimises $D_{\mathrm{KL}}(p_{\text{data}} \parallel p_\theta)$.

• Latent variable models: $p_\theta(x) = \int p_\theta(x \mid z) p(z) dz$. This is the law of total probability (Week 4) applied continuously. The latent $z$ is a random variable (Week 2); the integral marginalises it out.

• ELBO (Evidence Lower Bound): Computing the marginal likelihood is intractable. Variational inference replaces it with a bound:

  $$\log p_\theta(x) \ge \mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)] - D_{\mathrm{KL}}(q_\phi(z \mid x) \parallel p(z)).$$

  This uses: conditional expectation (Week 3), KL divergence (Week 11), and Bayes-flavoured inversion ($q(z \mid x)$ approximates the posterior $p(z \mid x)$, Week 5/9).

• Diffusion models: The forward process adds Gaussian noise (Week 6 — multivariate Gaussian), and the reverse process learns to denoise. The training objective is a weighted sum of denoising score-matching losses, derived from a variational bound on the KL between the data and model distributions (Week 11).

• Normalising flows: Transform a simple base distribution (e.g., standard Gaussian) through a sequence of invertible maps. The change-of-variables formula for densities uses the determinant of the Jacobian — Linear Algebra (matrix factorisations from LA) meets probability.

The Generative Models course (course index) builds these derivations step by step. You now have the probability vocabulary to follow them.

Case Prompts#

For each scenario below, identify which probabilistic tool from this course is the best diagnostic or solution approach.

Case 1 — Rare-disease screening. A classifier for a rare disease (base rate 0.1%) has 99% accuracy. The product manager is confused: "If the test is 99% accurate, why are most positive predictions wrong?"

Tool: Bayes' rule and base-rate analysis (Week 5). Compute $P(\text{disease} \mid \text{positive})$ using the prior (0.001) and the false-positive rate. The vast majority of positives are false positives because the disease is so rare — exactly the medical-test example from Week 5.

Case 2 — Overconfident regression. Your uncertainty-aware regression model outputs Gaussian predictive distributions $\mathcal{N}(\mu(x), \sigma^2(x))$. On held-out data, the 95% prediction intervals contain the true value only 70% of the time.

Tool: Calibration and sharpness analysis. Check if $\sigma(x)$ is systematically too small (underestimated uncertainty). This is a bias–variance misspecification (Week 10): the model's variance estimate is biased downward. Could also be a model misspecification — the true conditional distribution may be heavier-tailed than Gaussian (Week 6 limitations).

Case 3 — Mode collapse in a GAN. Your generator produces sharp but limited-diversity images — it seems to have locked onto a subset of the data distribution.

Tool: KL direction analysis (Week 11). The standard GAN loss approximates reverse KL (or Jensen-Shannon), which is mode-seeking. Mode collapse is the expected behaviour — the generator finds a few high-quality modes and ignores the rest. Forward KL (MLE) would penalise missed modes, but GANs optimise a different divergence.

Mistake Checklist#

Consolidate the warnings from across the course into a single auditing checklist. When you build or review a probabilistic model, ask:

1. Base-rate neglect (Week 5): Did you account for the prior probability of the event? A "99% accurate" test for a rare condition still produces mostly false positives.

2. PDF is not probability (Week 2): For continuous variables, $f(x) = 2.5$ is not a probability — it is a density. Probability comes from integration. A PDF can exceed 1.

3. Uncorrelated $\ne$ independent (Week 3): $Y = X^2$ with symmetric $X$ has $\mathrm{Cov}(X, Y) = 0$ but perfect dependence. Correlation only captures linear relationships.

4. CI misinterpretation (Week 10): A 95% confidence interval does not mean "95% probability the parameter is in this interval." It means "95% of intervals constructed this way would contain the parameter."

5. KL direction matters (Week 11): $D_{\mathrm{KL}}(p \parallel q) \ne D_{\mathrm{KL}}(q \parallel p)$. Forward KL penalises $q$ for ignoring data modes (mode-covering). Reverse KL penalises $q$ for venturing where there is no data (mode-seeking). Know which one your loss implements.

6. CLT misuse (Week 7): The CLT requires finite variance and sufficiently large $n$. Heavy-tailed data (e.g., Cauchy) or very small $n$ with skewed populations break the normal approximation.

7. Conditioning on colliders (Week 13): Selecting samples based on an outcome (e.g., analysing only successful startups) induces spurious correlations between otherwise independent causes. This is selection bias — and it is everywhere in observational data.

8. Using point estimates when uncertainty matters: MLE and MAP give you a single $\theta$. If your downstream decision is sensitive to parameter uncertainty (medical treatment, financial risk), use the full posterior or at least bootstrap confidence intervals.

Knowledge check#

Browser lab#

Two mini demos: (1) compute the value function for a 2-state MRP by linear expectation backup, and (2) compute KL between an empirical histogram of samples and a misspecified categorical model.

import numpy as np
import matplotlib.pyplot as plt

# Part 1: 2-state MRP value computation
# States: 0 and 1. Discount γ = 0.9.
# P(s' | s): P[0,:] = [0.7, 0.3], P[1,:] = [0.4, 0.6]
# Rewards: R[0]=1, R[1]=2

gamma = 0.9
P = np.array([[0.7, 0.3],
               [0.4, 0.6]])
R = np.array([1.0, 2.0])

# Solve Bellman: V = R + γ P V  =>  (I - γP)V = R
I = np.eye(2)
V = np.linalg.solve(I - gamma * P, R)

print("Part 1 — 2-state MRP")
print(f"  Transition matrix P:\n{P}")
print(f"  Rewards R: {R}")
print(f"  Value function V: {V}")

# Part 2: KL between empirical histogram and misspecified model
rng = np.random.default_rng(99)
n = 1000
true_p = np.array([0.6, 0.25, 0.1, 0.05])
samples = rng.choice(4, size=n, p=true_p)

empirical_p = np.bincount(samples, minlength=4) / n

misspecified_q = np.array([0.25, 0.25, 0.25, 0.25])

def kl_div(p, q):
    mask = p > 0
    return np.sum(p[mask] * np.log(p[mask] / q[mask]))

kl_val = kl_div(empirical_p, misspecified_q)
kl_true = kl_div(true_p, misspecified_q)

print(f"\nPart 2 — KL divergence")
print(f"  True p:        {true_p}")
print(f"  Empirical p̂:   {np.round(empirical_p, 4)}")
print(f"  Misspecified q: {misspecified_q}")
print(f"  KL(p̂ || q) = {kl_val:.4f}  (KL from empirical)")
print(f"  KL(p || q)  = {kl_true:.4f}  (KL from true)")

fig, axes = plt.subplots(1, 2, figsize=(10, 4.5))

axes[0].bar(["State 0", "State 1"], V, color=["tab:blue", "tab:orange"])
axes[0].set_title("Value Function V(s)")
axes[0].set_ylabel("Value")

x = np.arange(4)
w = 0.3
axes[1].bar(x - w, true_p, w, label="True p", alpha=0.8)
axes[1].bar(x, empirical_p, w, label="Empirical p̂", alpha=0.8)
axes[1].bar(x + w, misspecified_q, w, label="Misspecified q", alpha=0.8)
axes[1].set_title("Distribution Comparison")
axes[1].set_xticks(x)
axes[1].legend()
plt.tight_layout()
plt.show()