Week 12: Exponential Families (lite)

One Form to Rule Them#

Many of the most important distributions in ML — Bernoulli, Binomial, Poisson, Gaussian, Beta, Dirichlet, Categorical, Gamma — share a common algebraic structure called the exponential family:

The four components:

| Symbol | Name | Role | |---|---|---| | $\eta$ | Natural parameter (canonical parameter) | The parameter in the "natural" coordinate system | | $T(x)$ | Sufficient statistic | Summarises all information in $x$ needed to estimate $\eta$ | | $A(\eta)$ | Log-partition function (log-normaliser) | Ensures the density integrates/sums to 1 | | $h(x)$ | Base measure | The part that does not depend on $\eta$ |

The log-partition is not arbitrary — it is determined by the requirement that $p$ integrates to 1:

Why exponential families matter in ML:

• MLE is convex in $\eta$: the log-likelihood $\eta \cdot \sum T(x_i) - n A(\eta)$ is concave in $\eta$, so finding the MLE is a convex optimisation problem.
• Sufficient statistics of fixed size: regardless of sample size $n$, you only need the $d$-dimensional vector $\sum T(x_i)$ to compute the MLE.
• Conjugate priors exist naturally in the same family.
• Maximum entropy property: among all distributions with a given expected sufficient statistic $\mathbb{E}[T(X)]$, the exponential family member has maximum entropy.

Bernoulli and Gaussian as Examples#

Bernoulli in exponential family form#

The Bernoulli PMF is $p(x) = p^x (1-p)^{1-x}$ for $x \in \{0, 1\}$. Rewrite:

Read off the components:

• Natural parameter: $\eta = \log\frac{p}{1-p}$ (the log-odds or logit).
• Sufficient statistic: $T(x) = x$.
• Log-partition: $A(\eta) = -\log(1-p) = \log(1 + e^\eta)$.
• Base measure: $h(x) = 1$.

The log-partition $A(\eta) = \log(1 + e^\eta)$ is the softplus function — this is exactly the activation that appears in the binary cross-entropy loss.

Mean parameterisation: The mean is $\mu = \mathbb{E}[X] = p$. The relationship between natural parameter and mean is:

Indeed, in GLMs and logistic regression, the sigmoid maps the linear predictor (natural parameter space) to the probability (mean parameter space). This is not a coincidence — it is the exponential family structure.

Gaussian (known variance) in exponential family form#

For $X \sim \mathcal{N}(\mu, \sigma^2)$ with known $\sigma^2$:

Components:

• $\eta = \mu / \sigma^2$
• $T(x) = x$
• $A(\eta) = \mu^2 / (2\sigma^2) = \eta^2 \sigma^2 / 2$
• $h(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp(-x^2 / (2\sigma^2))$

The mean parameter is $A'(\eta) = \eta \sigma^2 = \mu$ (the identity link in linear regression).

Log-Partition and Moments#

A key property: derivatives of $A(\eta)$ give moments of the sufficient statistic:

The gradient of the log-partition is the mean of $T(X)$; the Hessian is its covariance. This follows from differentiating under the integral sign in the definition of $A(\eta)$.

Worked check — Bernoulli. $A(\eta) = \log(1 + e^\eta)$. Then:

For maximum likelihood: Setting $\nabla_\eta \ell(\eta) = \sum T(x_i) - n \nabla A(\eta) = 0$ gives:

The MLE matches the theoretical moments of $T(X)$ to the empirical moments of the data — moment matching. This is why MLE is often equivalent to solving $\mathbb{E}_\theta[T(X)] = \bar T_{\text{emp}}$.

Softmax / Categorical Pointer#

The Categorical distribution (multi-class Bernoulli) over $k$ classes is also in the exponential family. The natural parameters form a $(k-1)$-dimensional vector (one class is the reference). The log-partition is the log-sum-exp:

Its gradient is the softmax function — again, the activation function is determined by the exponential family structure. Multi-class cross-entropy loss with softmax output is exactly the negative log-likelihood of a categorical exponential family model.

Why ML Cares#

The exponential family scaffolding runs through much of modern ML:

• Generalised Linear Models (GLMs): linear regression (Gaussian), logistic regression (Bernoulli), Poisson regression — all assume the response distribution is in an exponential family with a linear predictor in natural parameter space.
• Variational inference: the mean-field update equations are particularly simple for conditionally conjugate exponential family models.
• Maximum entropy principle: the exponential family is the unique max-entropy distribution for a given expected sufficient statistic — used in inverse RL and maximum entropy RL.
• Information geometry: the Fisher information matrix is exactly $\nabla^2 A(\eta)$, and the natural gradient uses it to re-scale gradient steps.

This week is a "unification lens" — not a full GLM or information geometry course. The goal is to recognise the common structure so that when you see softmax, sigmoid, or moment-matching equations in research papers, you know where they come from.

Knowledge check#

Browser lab#

Plot $A(\eta)$ and its derivative (mean parameter vs natural parameter) for the Bernoulli, showing the sigmoid link.

import numpy as np
import matplotlib.pyplot as plt

eta = np.linspace(-5, 5, 300)

A = np.log(1 + np.exp(eta))
dA = 1 / (1 + np.exp(-eta))
d2A = dA * (1 - dA)

fig, axes = plt.subplots(1, 3, figsize=(14, 4))

axes[0].plot(eta, A, "tab:blue", linewidth=2)
axes[0].set_title("Log-partition A(η) = log(1 + e^η)")
axes[0].set_xlabel("η (log-odds)"); axes[0].set_ylabel("A(η)")

axes[1].plot(eta, dA, "tab:orange", linewidth=2)
axes[1].set_title("Mean μ = A'(η) = σ(η)")
axes[1].set_xlabel("η (log-odds)"); axes[1].set_ylabel("μ (probability)")
axes[1].set_ylim(-0.05, 1.05)

axes[2].plot(eta, d2A, "tab:green", linewidth=2)
axes[2].set_title("Variance A''(η) = μ(1−μ)")
axes[2].set_xlabel("η (log-odds)"); axes[2].set_ylabel("Var")
axes[2].axhline(0.25, color="gray", linestyle="--", alpha=0.5, label="max=0.25")
axes[2].legend()

plt.tight_layout()
plt.show()

# Demonstrate MLE via moment matching
# Generate Bernoulli data with p=0.7
rng = np.random.default_rng(123)
n = 200
p_true = 0.7
data = rng.binomial(1, p_true, size=n)

p_hat = data.mean()
eta_hat = np.log(p_hat / (1 - p_hat))

print(f"True p = {p_true}, MLE p̂ = {p_hat:.4f}, η̂ = {eta_hat:.4f}")
print(f"A'(η̂) = {1/(1+np.exp(-eta_hat)):.4f} ≈ p̂ — moment matching confirmed")