Week 7: Laws of Large Numbers and the CLT

i.i.d. samples#

The most important assumption in classical statistics: independent and identically distributed (i.i.d.). A sequence $X_1, X_2, \ldots, X_n$ is i.i.d. if:

• Identically distributed: each $X_i$ has the same CDF/PMF/PDF, so they are draws from the same population.
• Independent: knowing $X_1, \ldots, X_{i-1}$ tells you nothing about $X_i$.

Think of repeatedly rolling the same die, or drawing with replacement from a population. The i.i.d. assumption makes the sample mean a simple, well-behaved statistic.

Why it matters: both the Law of Large Numbers and the Central Limit Theorem require i.i.d. (or at least weak dependence + identical marginals). When data are dependent (time series, spatial data, within-subject repeats), the standard formulas underestimate uncertainty.

Law of large numbers#

The Weak Law of Large Numbers (LLN): as $n \to \infty$, the sample mean $\bar X_n = \frac{1}{n}\sum_{i=1}^n X_i$ converges in probability to the true mean $\mu = \mathbb{E}[X_i]$:

In plain language: with enough samples, the sample average is almost certainly close to the population mean. The probability of a deviation larger than any fixed $\varepsilon$ shrinks to zero.

Why convergence in probability, not convergence of the sequence? The $\bar X_n$ are random: each repetition of the experiment gives a different path. The LLN says that the probability of being far from $\mu$ vanishes; it does not say that a particular realised sequence must converge. (The Strong Law does, but we use the Weak Law here as it is sufficient for our purposes.)

Monte Carlo integration is a direct application: to estimate $\mathbb{E}[g(X)]$, sample $X_1, \ldots, X_n \sim p$ and compute $\frac{1}{n}\sum_i g(X_i)$. The LLN guarantees this converges to the true expectation. This is how many Bayesian computations and reinforcement learning value estimates work.

Worked example: coin flips. Flip a fair coin $n$ times; let $X_i = 1$ for heads, 0 for tails. $\mathbb{E}[X_i] = 0.5$. The sample proportion of heads converges to 0.5:

• $n = 10$: $\bar X$ might be anywhere from 0.2 to 0.8.
• $n = 100$: usually within 0.4–0.6.
• $n = 10,000$: almost always within 0.49–0.51.

The LLN tells us that concentration happens, but not how fast. That is the role of the CLT and variance formulas.

Central limit theorem#

The Central Limit Theorem (CLT) is deeper: it describes the shape of the sampling distribution of the mean. For i.i.d. $X_i$ with mean $\mu$ and finite variance $\sigma^2$:

"$\xrightarrow{d}$" means convergence in distribution: the CDF of the standardised mean approaches the standard normal CDF.

What the CLT says:

• The sample mean $\bar X_n$ is approximately $\mathcal{N}(\mu, \sigma^2 / n)$.
• The approximation improves as $n$ grows.
• The original distribution of $X_i$ can be anything (discrete, skewed, bounded, multimodal): as long as the variance is finite, sums become Gaussian.
• The rate of convergence is roughly $O(1/\sqrt{n})$ (Berry–Esseen theorem).

CLT in practice: standard error. The standard deviation of $\bar X_n$ is $\sigma / \sqrt{n}$, called the standard error. It shrinks at rate $1 / \sqrt{n}$, so to halve your error, you need four times as many samples.

Worked example: exponential population. $X_i \sim \text{Exponential}(\lambda = 1)$. Then $\mu = 1$, $\sigma^2 = 1$. For $n = 100$, the standard error is $\sigma / \sqrt{n} = 1/10 = 0.1$. So $\bar X_{100}$ is approximately $\mathcal{N}(1, 0.1^2)$. About 95% of sample means will fall in $\mu \pm 2 \cdot \text{SE} = 1 \pm 0.2$, or $[0.8, 1.2]$.

Notice that the original exponential data is far from normal: it is skewed right, with a sharp peak at 0. Yet the distribution of the mean of 100 such observations is nearly bell-shaped. That is the CLT's power.

Sampling distribution of the mean#

The sampling distribution of a statistic is the distribution of that statistic over repeated samples of size $n$ from the same population.

For the sample mean:

• Mean of sampling distribution: $\mathbb{E}[\bar X_n] = \mu$ (unbiased).
• Variance of sampling distribution: $\mathrm{Var}(\bar X_n) = \sigma^2 / n$ (shrinks with $n$).
• Shape: approximately normal for large $n$ (CLT), exactly normal for any $n$ if the $X_i$ are themselves normal.

Degrees-of-freedom intuition: why $\sigma^2 / n$? When we average $n$ independent observations, their individual fluctuations partly cancel. The variance of a sum of independent variables is the sum of variances: $\mathrm{Var}(\sum X_i) = n\sigma^2$. Dividing by $n$ gives $\mathrm{Var}(\bar X_n) = \sigma^2 / n^2 \cdot n = \sigma^2 / n$.

Limitations#

The CLT is powerful but not magical. It fails or misleads in important cases:

• Heavy tails (infinite variance): If $\sigma^2 = \infty$ (e.g., Cauchy distribution), the CLT does not apply. Sums of Cauchy variables remain Cauchy, not Gaussian, so the sample mean never concentrates.
• Small $n$ with skewed data: For $n = 5$, the sampling distribution of the mean from an exponential population is still noticeably skewed. The $n \to \infty$ guarantee does not say what happens at $n = 5$.
• Dependence: If $X_i$ are not independent (e.g., stock returns on consecutive days), the standard error $\sigma/\sqrt{n}$ is wrong (usually an underestimate). Time series methods (autocorrelation, block bootstrap) are needed.
• Extreme quantiles: The CLT describes the centre of the distribution, not the tails. For a 99.9% VaR calculation, normality of the mean is insufficient, since tail behaviour depends on the original distribution's tail.

The pragmatic rule: for "nice" distributions (finite variance, not pathologically skewed) with $n \ge 30$, the normal approximation is usually adequate for central intervals. For confidence intervals on the mean, the CLT is the workhorse, but always check your data's tail behaviour before trusting extreme quantiles.

Knowledge check#

Browser lab#

Show the LLN (running mean vs n) and CLT (histogram of many sample means) for uniform and exponential populations.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(42)
pop_size = 5000
n_means = 2000

fig, axes = plt.subplots(2, 3, figsize=(14, 8))

for row, (dist_name, dist_fn, mu_true, sigma_true) in enumerate([
    ("Uniform(2, 6)", lambda sz: rng.uniform(2, 6, size=sz), 4.0, np.sqrt(16/12)),
    ("Exponential(λ=1)", lambda sz: rng.exponential(1, size=sz), 1.0, 1.0),
]):
    # LLN: single long sequence
    samples = dist_fn(pop_size)
    running_mean = np.cumsum(samples) / np.arange(1, pop_size + 1)
    ax = axes[row, 0]
    ax.plot(running_mean, linewidth=0.8)
    ax.axhline(mu_true, color="tab:red", linestyle="--", label=f"μ = {mu_true}")
    ax.set_xscale("log"); ax.set_xlabel("n (log scale)")
    ax.set_ylabel("Running mean")
    ax.set_title(f"LLN — {dist_name}")
    ax.legend()

    # CLT: many sample means for a fixed n
    for col_idx, n in enumerate([5, 30]):
        means = np.array([dist_fn(n).mean() for _ in range(n_means)])
        ax = axes[row, 1 + col_idx]
        ax.hist(means, bins=50, density=True, alpha=0.7)
        x_grid = np.linspace(means.min(), means.max(), 200)
        se = sigma_true / np.sqrt(n)
        normal_approx = (1/(np.sqrt(2*np.pi)*se)) * np.exp(-0.5*((x_grid - mu_true)/se)**2)
        ax.plot(x_grid, normal_approx, "tab:red", linewidth=2, label="CLT approx")
        ax.set_title(f"CLT — n={n}, {dist_name}")
        ax.set_xlabel("Sample mean"); ax.legend()

plt.tight_layout()
plt.show()

print("LLN + CLT demo complete.")
print("Uniform: μ=4.0, σ/√30 ≈ {:.3f}".format(np.sqrt(16/12)/np.sqrt(30)))
print("Exponential: μ=1.0, σ/√30 ≈ {:.3f}".format(1.0/np.sqrt(30)))

print("Notice: the standardised sample means converge to σ/√30 ≈ 0.211 (uniform) and 0.183 (exponential), whatever the shape of the underlying draws.")

Key takeaways#

• The i.i.d. assumption (identical distribution and independence) is what makes the sample mean well-behaved. Dependence breaks the standard error formulas.
• The weak law of large numbers says the sample mean converges in probability to the true mean. This is what licenses Monte Carlo estimation of expectations.
• The central limit theorem says the standardised sample mean converges in distribution to a standard normal, whatever the shape of the population, as long as the variance is finite.
• The standard error is $\sigma / \sqrt{n}$, so halving the error requires four times the data.
• The CLT fails for infinite-variance distributions, is unreliable at small $n$ with skewed data, and says nothing about the tails.
• For central intervals with $n$ around 30 or more and finite variance, the normal approximation is usually adequate; check the tails before trusting extreme quantiles.

Further reading#

• Joseph K. Blitzstein & Jessica Hwang, Introduction to Probability (2nd ed., CRC Press, 2019): the law of large numbers and central limit theorem with simulation-based intuition.
• Larry Wasserman, All of Statistics (Springer, 2004): Chapter 5 covers convergence in probability and distribution.
• George Casella & Roger L. Berger, Statistical Inference (2nd ed., Duxbury, 2002): the limit-theorem chapter with full proofs.