Week 3: Expectation, Variance, and Covariance

Expectation#

The expectation (expected value, mean) of a random variable $X$ is the probability-weighted average of its possible values:

Discrete: $\displaystyle \mathbb{E}[X] = \sum_x x \cdot p_X(x)$

Continuous: $\displaystyle \mathbb{E}[X] = \int_{-\infty}^{\infty} x \cdot f_X(x)\, dx$

Expectation is the centre of mass of the distribution, representing the balancing point if you placed probability mass along a number line.

Worked examples:

• Bernoulli$(p)$: $\mathbb{E}[X] = 0 \cdot (1-p) + 1 \cdot p = p$.
• Binomial$(n, p)$: $\mathbb{E}[X] = np$ (derivable from the sum-of-Bernoullis view).
• Uniform$(a, b)$: $\mathbb{E}[X] = \frac{a+b}{2}$ (the midpoint).
• Exponential$(\lambda)$: $\mathbb{E}[X] = 1 / \lambda$ (rate $=$ inverse of mean wait time).

LOTUS (Law of the Unconscious Statistician): To find $\mathbb{E}[g(X)]$, you do not need the distribution of $g(X)$. You integrate or sum $g(x)$ against the distribution of $X$:

Worked example: $\mathbb{E}[X^2]$ for a fair die. With $p(x) = 1/6$ for $x = 1,\ldots,6$:

Contrast this with $(\mathbb{E}[X])^2 = (3.5)^2 = 12.25$; the expectation of the square is larger than the square of the expectation (Jensen's inequality).

Linearity of expectation#

Linearity is the most powerful shortcut in probability. For any random variables $X, Y$ and constants $a, b$:

No independence required. The expectation of a sum is always the sum of expectations, regardless of whether $X$ and $Y$ are independent, correlated, or even deterministically linked.

Proof sketch (discrete):

Worked example: dependent variables. Let $X$ be the roll of a die, and $Y = 7 - X$ (so $Y$ is completely determined by $X$). Then $\mathbb{E}[X] = 3.5$, $\mathbb{E}[Y] = 3.5$, and

Indeed $X + Y = 7$ always, so $\mathbb{E}[7] = 7$. Linearity works despite perfect dependence.

Binomial expectation via linearity: $X = \sum_{i=1}^n Y_i$ where $Y_i \sim \text{Bernoulli}(p)$. Even if the $Y_i$ were correlated, $\mathbb{E}[X] = \sum \mathbb{E}[Y_i] = np$ (for i.i.d. they happen to be independent, but linearity does not need that).

Variance#

The variance measures spread: how far $X$ tends to be from its mean:

The standard deviation is $\sigma_X = \sqrt{\mathrm{Var}(X)}$, which has the same units as $X$.

Computational formula (often easier to calculate):

Proof: $\mathbb{E}[(X - \mu)^2] = \mathbb{E}[X^2 - 2\mu X + \mu^2] = \mathbb{E}[X^2] - 2\mu\,\mathbb{E}[X] + \mu^2 = \mathbb{E}[X^2] - \mu^2$.

Scaling and shifting:

The constant shift $b$ adds no variance. Scaling by $a$ scales variance by $a^2$ (since variance is in squared units).

Worked examples:

• Bernoulli$(p)$: $\mathrm{Var}(X) = p(1-p)$. Maximum at $p = 0.5$.
• Binomial$(n, p)$: $\mathrm{Var}(X) = np(1-p)$.
• Uniform$(a, b)$: $\mathrm{Var}(X) = \frac{(b-a)^2}{12}$.
• Exponential$(\lambda)$: $\mathrm{Var}(X) = 1 / \lambda^2$.

Worked computation: fair die. $\mathbb{E}[X] = 3.5$. From earlier, $\mathbb{E}[X^2] = 91/6 \approx 15.167$. So

Standard deviation $\sigma \approx 1.71$.

Covariance and correlation#

Covariance measures how two variables move together:

Computational form: $\mathrm{Cov}(X, Y) = \mathbb{E}[XY] - \mathbb{E}[X]\,\mathbb{E}[Y]$.

• Positive covariance: $X$ above its mean tends to happen with $Y$ above its mean.
• Negative covariance: opposite directions.
• Zero covariance: no linear relationship detected.

Properties:

• $\mathrm{Cov}(X, X) = \mathrm{Var}(X)$.
• $\mathrm{Cov}(X, Y) = \mathrm{Cov}(Y, X)$ (symmetric).
• $\mathrm{Cov}(aX + b, cY + d) = ac \, \mathrm{Cov}(X, Y)$.
• $\mathrm{Var}(X + Y) = \mathrm{Var}(X) + \mathrm{Var}(Y) + 2\,\mathrm{Cov}(X, Y)$.

The last formula shows why the variance of a sum is not always the sum of variances: the covariance term can add or subtract.

Correlation standardises covariance to the range $[-1, 1]$:

• $\rho = 1$: perfect positive linear relationship.
• $\rho = -1$: perfect negative linear relationship.
• $\rho = 0$: uncorrelated.

Critical warning: uncorrelated $\ne$ independent. Independence implies zero correlation, but zero correlation does not imply independence. For example, let $X \sim \text{Uniform}(-1, 1)$ and $Y = X^2$. Then $\mathrm{Cov}(X, Y) = \mathbb{E}[X^3] - \mathbb{E}[X]\mathbb{E}[X^2] = 0 - 0 \cdot (1/3) = 0$. They are uncorrelated but perfectly dependent: knowing $X$ determines $Y$ exactly.

Correlation only captures linear dependence. Non-linear relationships (quadratic, periodic) can have zero correlation but strong dependence.

Worked example: joint table. Suppose $X$ and $Y$ have the following joint PMF:

Then $\mathbb{E}[X] = 0.5$, $\mathbb{E}[Y] = 0.7$, $\mathbb{E}[XY] = 0 \cdot 0 \cdot 0.2 + 0 \cdot 1 \cdot 0.3 + 1 \cdot 0 \cdot 0.1 + 1 \cdot 1 \cdot 0.4 = 0.4$.

Small positive covariance: $X = 1$ slightly increases the chance of $Y = 1$.

Conditional expectation intro#

The conditional expectation $\mathbb{E}[X \mid Y = y]$ is the average of $X$ over the subset of outcomes where $Y = y$. It is a function of $y$: for each possible value of $Y$, we get a (potentially different) expected value of $X$.

Worked example: from the table above:

When $Y = 0$, $X$ averages lower; when $Y = 1$, $X$ averages higher, consistent with the positive covariance we found.

Law of total expectation: $\mathbb{E}[X] = \mathbb{E}_Y[\mathbb{E}[X \mid Y]]$: the overall mean is the weighted average of conditional means. Week 4 deepens this.

Knowledge check#

Browser lab#

Sample pairs from a known joint distribution, compute empirical mean/variance/covariance, and compare to formulas.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(42)
n = 5000

joint = np.array([[0.1, 0.2, 0.1],
                   [0.05, 0.3, 0.05],
                   [0.0, 0.1, 0.1]])
x_vals = np.array([0, 1, 2])
y_vals = np.array([0, 1, 2])

flat = joint.ravel() / joint.sum()
indices = rng.choice(len(flat), size=n, p=flat)
xi = indices // 3
yi = indices % 3
xs = x_vals[xi]
ys = y_vals[yi]

E_x = np.sum(x_vals * joint.sum(axis=1))
E_y = np.sum(y_vals * joint.sum(axis=0))
E_xy = np.sum(np.outer(x_vals, y_vals) * joint)
Cov_xy = E_xy - E_x * E_y

E_x_emp = xs.mean()
E_y_emp = ys.mean()
Cov_emp = np.mean((xs - E_x_emp) * (ys - E_y_emp))

print(f"E[X]  true={E_x:.4f}  emp={E_x_emp:.4f}")
print(f"E[Y]  true={E_y:.4f}  emp={E_y_emp:.4f}")
print(f"Cov   true={Cov_xy:.4f}  emp={Cov_emp:.4f}")

px = joint.sum(axis=1)
py = joint.sum(axis=0)

fig, axes = plt.subplots(1, 3, figsize=(13, 4))
axes[0].bar(x_vals, px, color="tab:blue", alpha=0.7)
axes[0].set_title("Marginal P(X)")
axes[0].set_xlabel("x")
axes[1].bar(y_vals, py, color="tab:orange", alpha=0.7)
axes[1].set_title("Marginal P(Y)")
axes[1].set_xlabel("y")
axes[2].scatter(xs[:500], ys[:500], alpha=0.3, s=8)
axes[2].set_title("Joint samples (first 500)")
axes[2].set_xlabel("x"); axes[2].set_ylabel("y")
plt.tight_layout()
plt.show()

print("Notice: sample E[X], E[Y] and Cov match the exact table (0.7922/0.8000, 1.0956/1.1000, 0.1205/0.1200) — the joint sample carries the dependence.")

Key takeaways#

• Expectation is the probability-weighted average, representing the centre of mass. LOTUS computes $\mathbb{E}[g(X)]$ directly against the distribution of $X$, with no need for the distribution of $g(X)$.
• Linearity of expectation holds for any random variables, dependent or not. That is why it is the most powerful shortcut in probability.
• Variance measures spread, with the shift-and-scale rule $\mathrm{Var}(aX + b) = a^2 \mathrm{Var}(X)$ and the computational form $\mathbb{E}[X^2] - (\mathbb{E}[X])^2$.
• Covariance measures linear co-movement; correlation standardises it to $[-1, 1]$.
• Uncorrelated does not imply independent. With symmetric $X$ and $Y = X^2$, the variables are perfectly dependent yet have zero covariance.
• $\mathrm{Var}(X + Y) = \mathrm{Var}(X) + \mathrm{Var}(Y) + 2\,\mathrm{Cov}(X, Y)$: the covariance term is why variances do not simply add.

Further reading#

• Joseph K. Blitzstein & Jessica Hwang, Introduction to Probability (2nd ed., CRC Press, 2019): expectation, variance, and covariance with strong intuition for linearity.
• George Casella & Roger L. Berger, Statistical Inference (2nd ed., Duxbury, 2002): Chapter 4 develops moments systematically.
• Larry Wasserman, All of Statistics (Springer, 2004): the expectation and variance sections are concise and applied.