Week 7: Laws of Large Numbers and the CLT

i.i.d. Samples#

The most important assumption in classical statistics: independent and identically distributed (i.i.d.). A sequence $X_1, X_2, \ldots, X_n$ is i.i.d. if:

• Identically distributed: each $X_i$ has the same CDF/PMF/PDF — they are draws from the same population.
• Independent: knowing $X_1, \ldots, X_{i-1}$ tells you nothing about $X_i$.

Think of repeatedly rolling the same die, or drawing with replacement from a population. The i.i.d. assumption makes the sample mean a simple, well-behaved statistic.

Why it matters: both the Law of Large Numbers and the Central Limit Theorem require i.i.d. (or at least weak dependence + identical marginals). When data are dependent (time series, spatial data, within-subject repeats), the standard formulas underestimate uncertainty.

Law of Large Numbers#

The Weak Law of Large Numbers (LLN): as $n \to \infty$, the sample mean $\bar X_n = \frac{1}{n}\sum_{i=1}^n X_i$ converges in probability to the true mean $\mu = \mathbb{E}[X_i]$:

In plain language: with enough samples, the sample average is almost certainly close to the population mean. The probability of a deviation larger than any fixed $\varepsilon$ shrinks to zero.

Why convergence in probability, not convergence of the sequence? The $\bar X_n$ are random — each repetition of the experiment gives a different path. The LLN says that the probability of being far from $\mu$ vanishes; it does not say that a particular realised sequence must converge. (The Strong Law does, but we use the Weak Law here as it is sufficient for our purposes.)

Monte Carlo integration is a direct application: to estimate $\mathbb{E}[g(X)]$, sample $X_1, \ldots, X_n \sim p$ and compute $\frac{1}{n}\sum_i g(X_i)$. The LLN guarantees this converges to the true expectation. This is how many Bayesian computations and reinforcement learning value estimates work.

Worked example — coin flips. Flip a fair coin $n$ times; let $X_i = 1$ for heads, 0 for tails. $\mathbb{E}[X_i] = 0.5$. The sample proportion of heads converges to 0.5:

• $n = 10$: $\bar X$ might be anywhere from 0.2 to 0.8.
• $n = 100$: usually within 0.4–0.6.
• $n = 10,000$: almost always within 0.49–0.51.

The LLN tells us that concentration happens, but not how fast. That is the role of the CLT and variance formulas.

Central Limit Theorem#

The Central Limit Theorem (CLT) is deeper: it describes the shape of the sampling distribution of the mean. For i.i.d. $X_i$ with mean $\mu$ and finite variance $\sigma^2$:

"$\xrightarrow{d}$" means convergence in distribution: the CDF of the standardised mean approaches the standard normal CDF.

What the CLT says:

• The sample mean $\bar X_n$ is approximately $\mathcal{N}(\mu, \sigma^2 / n)$.
• The approximation improves as $n$ grows.
• The original distribution of $X_i$ can be anything (discrete, skewed, bounded, multimodal) — as long as the variance is finite, sums become Gaussian.
• The rate of convergence is roughly $O(1/\sqrt{n})$ (Berry–Esseen theorem).

CLT in practice — standard error. The standard deviation of $\bar X_n$ is $\sigma / \sqrt{n}$, called the standard error. It shrinks at rate $1 / \sqrt{n}$ — to halve your error, you need four times as many samples.

Worked example — Exponential population. $X_i \sim \text{Exponential}(\lambda = 1)$. Then $\mu = 1$, $\sigma^2 = 1$. For $n = 100$, the standard error is $\sigma / \sqrt{n} = 1/10 = 0.1$. So $\bar X_{100}$ is approximately $\mathcal{N}(1, 0.1^2)$. About 95% of sample means will fall in $\mu \pm 2 \cdot \text{SE} = 1 \pm 0.2$, or $[0.8, 1.2]$.

Notice that the original exponential data is far from normal — it is skewed right, with a sharp peak at 0. Yet the distribution of the mean of 100 such observations is nearly bell-shaped. That is the CLT's power.

Sampling Distribution of the Mean#

The sampling distribution of a statistic is the distribution of that statistic over repeated samples of size $n$ from the same population.

For the sample mean:

• Mean of sampling distribution: $\mathbb{E}[\bar X_n] = \mu$ (unbiased).
• Variance of sampling distribution: $\mathrm{Var}(\bar X_n) = \sigma^2 / n$ (shrinks with $n$).
• Shape: approximately normal for large $n$ (CLT), exactly normal for any $n$ if the $X_i$ are themselves normal.

Degrees-of-freedom intuition: why $\sigma^2 / n$? When we average $n$ independent observations, their individual fluctuations partly cancel. The variance of a sum of independent variables is the sum of variances: $\mathrm{Var}(\sum X_i) = n\sigma^2$. Dividing by $n$ gives $\mathrm{Var}(\bar X_n) = \sigma^2 / n^2 \cdot n = \sigma^2 / n$.

Limitations#

The CLT is powerful but not magical. It fails or misleads in important cases:

• Heavy tails (infinite variance): If $\sigma^2 = \infty$ (e.g., Cauchy distribution), the CLT does not apply. Sums of Cauchy variables remain Cauchy, not Gaussian — the sample mean never concentrates.
• Small $n$ with skewed data: For $n = 5$, the sampling distribution of the mean from an exponential population is still noticeably skewed. The $n \to \infty$ guarantee does not say what happens at $n = 5$.
• Dependence: If $X_i$ are not independent (e.g., stock returns on consecutive days), the standard error $\sigma/\sqrt{n}$ is wrong (usually an underestimate). Time series methods (autocorrelation, block bootstrap) are needed.
• Extreme quantiles: The CLT describes the centre of the distribution, not the tails. For a 99.9% VaR calculation, normality of the mean is insufficient — tail behaviour depends on the original distribution's tail.

The pragmatic rule: for "nice" distributions (finite variance, not pathologically skewed) with $n \ge 30$, the normal approximation is usually adequate for central intervals. For confidence intervals on the mean, the CLT is the workhorse — but always check your data's tail behaviour before trusting extreme quantiles.

Knowledge check#

Browser lab#

Show the LLN (running mean vs n) and CLT (histogram of many sample means) for uniform and exponential populations.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(42)
pop_size = 5000
n_means = 2000

fig, axes = plt.subplots(2, 3, figsize=(14, 8))

for row, (dist_name, dist_fn, mu_true, sigma_true) in enumerate([
    ("Uniform(2, 6)", lambda sz: rng.uniform(2, 6, size=sz), 4.0, np.sqrt(16/12)),
    ("Exponential(λ=1)", lambda sz: rng.exponential(1, size=sz), 1.0, 1.0),
]):
    # LLN: single long sequence
    samples = dist_fn(pop_size)
    running_mean = np.cumsum(samples) / np.arange(1, pop_size + 1)
    ax = axes[row, 0]
    ax.plot(running_mean, linewidth=0.8)
    ax.axhline(mu_true, color="tab:red", linestyle="--", label=f"μ = {mu_true}")
    ax.set_xscale("log"); ax.set_xlabel("n (log scale)")
    ax.set_ylabel("Running mean")
    ax.set_title(f"LLN — {dist_name}")
    ax.legend()

    # CLT: many sample means for a fixed n
    for col_idx, n in enumerate([5, 30]):
        means = np.array([dist_fn(n).mean() for _ in range(n_means)])
        ax = axes[row, 1 + col_idx]
        ax.hist(means, bins=50, density=True, alpha=0.7)
        x_grid = np.linspace(means.min(), means.max(), 200)
        se = sigma_true / np.sqrt(n)
        normal_approx = (1/(np.sqrt(2*np.pi)*se)) * np.exp(-0.5*((x_grid - mu_true)/se)**2)
        ax.plot(x_grid, normal_approx, "tab:red", linewidth=2, label="CLT approx")
        ax.set_title(f"CLT — n={n}, {dist_name}")
        ax.set_xlabel("Sample mean"); ax.legend()

plt.tight_layout()
plt.show()

print("LLN + CLT demo complete.")
print("Uniform: μ=4.0, σ/√30 ≈ {:.3f}".format(np.sqrt(16/12)/np.sqrt(30)))
print("Exponential: μ=1.0, σ/√30 ≈ {:.3f}".format(1.0/np.sqrt(30)))