Week 8: Point Estimation — Maximum Likelihood

Estimators as Functions of Data#

An estimator $\hat \theta = g(X_1, \ldots, X_n)$ is a function that maps a dataset to a guess about the unknown parameter $\theta$. It is a random variable (the data are random), and we judge it by its sampling distribution — bias, variance, and MSE (Week 10).

Example estimators for the mean of a population:

• Sample mean: $\hat \mu = \frac{1}{n}\sum X_i$ — standard choice.
• Sample median: robust to outliers, less efficient for normal data.
• Trimmed mean: discards extremes, a compromise.

The question is: which function of the data should we use? Maximum likelihood provides a principled answer: choose the $\theta$ that makes the observed data most probable.

Likelihood#

Given i.i.d. data $x_1, \ldots, x_n$, the likelihood function is the joint probability (density) viewed as a function of $\theta$:

The log-likelihood is almost always easier to work with (sums are friendlier than products, and the log is monotonic so the maximiser is unchanged):

Critical distinction — likelihood vs probability:

• $p(x \mid \theta)$ as a function of $x$ (with $\theta$ fixed) is a probability — it sums/integrates to 1 over $x$.
• $L(\theta) = p(x \mid \theta)$ as a function of $\theta$ (with $x$ fixed) is the likelihood — it does not integrate to 1 over $\theta$. "The likelihood of $\theta$" is not "the probability that $\theta$ is true."

The MLEMaximum Likelihood Estimation is the value that maximises the likelihood:

Since $\log$ is monotonic, maximising $\ell$ is equivalent to maximising $L$.

Bernoulli MLE#

Suppose we observe $n$ i.i.d. Bernoulli trials with unknown success probability $p$. The data are $x_1, \ldots, x_n \in \{0, 1\}$. Let $k = \sum x_i$ be the number of successes.

The likelihood of $p$ given the data:

Log-likelihood:

Set derivative to zero:

The MLE for a Bernoulli probability is simply the sample proportion of successes — intuitively the most natural estimate.

Worked numeric example. 7 heads in 10 coin flips. Then $\hat p = 0.7$. The log-likelihood at $\hat p = 0.7$ is:

At $p = 0.5$ (fair coin hypothesis), $\ell(0.5) = 7 \log(0.5) + 3 \log(0.5) = 10 \log(0.5) \approx -6.93$, which is lower — the data favour $p = 0.7$ over $p = 0.5$.

Gaussian MLE#

For i.i.d. data $x_1, \ldots, x_n$ assumed to come from $\mathcal{N}(\mu, \sigma^2)$ with both parameters unknown:

Joint density:

Log-likelihood:

MLE for $\mu$: Differentiate with respect to $\mu$ (the first two terms do not involve $\mu$):

MLE for $\sigma^2$: Plug $\hat \mu$ back and differentiate with respect to $\sigma^2$:

Note the denominator is $n$, not $n-1$. The MLE for variance is slightly biased downward (the $1/(n-1)$ version is unbiased — see Week 10). For large $n$, the difference is negligible.

MLE and Training Objectives#

MLE provides a unified lens on many ML training objectives:

• Linear regression with Gaussian noise: Assuming $y_i \sim \mathcal{N}(w^T x_i, \sigma^2)$, the negative log-likelihood is $\propto \sum (y_i - w^T x_i)^2$ — exactly the mean squared error loss.
• Logistic regression: Binary labels modelled as Bernoulli with $p = \sigma(w^T x)$, the negative log-likelihood is the binary cross-entropy loss.
• Multi-class classification: Categorical likelihood gives the cross-entropy loss with softmax.

The deep connection (preview Week 11): Maximising likelihood is equivalent to minimising the KL divergence between the empirical distribution of the data and the model distribution $p(x \mid \theta)$. Training a generative model by MLE means: make the model's distribution as close as possible (in KL sense) to the data distribution.

Invariance of MLE: If $\hat \theta$ is the MLE for $\theta$, then $g(\hat \theta)$ is the MLE for $g(\theta)$ for any one-to-one function $g$. Example: the MLE for standard deviation $\sigma$ is $\sqrt{\hat \sigma^2_{\text{MLE}}}$. This property does not hold for unbiased estimators in general.

Knowledge check#

Browser lab#

Plot the log-likelihood of Bernoulli $p$ for a fixed dataset and mark the MLE.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(7)
n, p_true = 30, 0.65
data = rng.binomial(1, p_true, size=n)
k = data.sum()

def log_lik(p):
    if p <= 0 or p >= 1:
        return -np.inf
    return k * np.log(p) + (n - k) * np.log(1 - p)

p_grid = np.linspace(0.01, 0.99, 200)
ll_vals = np.array([log_lik(p) for p in p_grid])
p_mle = k / n

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(p_grid, ll_vals, "tab:blue", linewidth=2, label="log-likelihood ℓ(p)")
ax.axvline(p_true, color="tab:green", linestyle="--", linewidth=1.5, label=f"True p={p_true}")
ax.axvline(p_mle, color="tab:red", linestyle="--", linewidth=1.5, label=f"MLE p̂={p_mle:.3f}")
ax.scatter([p_mle], [log_lik(p_mle)], color="tab:red", s=80, zorder=5)
ax.set_xlabel("p")
ax.set_ylabel("ℓ(p)")
ax.set_title(f"Log-likelihood of p — {k} heads in {n} flips")
ax.legend()
plt.tight_layout()
plt.show()

print(f"True p = {p_true}, observed k = {k}/{n}, MLE p̂ = {p_mle:.4f}")
print(f"ℓ(p̂) = {log_lik(p_mle):.3f}, ℓ(p_true) = {log_lik(p_true):.3f}")