Week 2: Random Variables and Distributions

Random Variables#

A random variable (RV) is a function $X: \Omega \to \mathbb{R}$ that assigns a real number to each outcome in the sample space. It is a bridge: the experiment produces an abstract outcome $\omega \in \Omega$, and $X(\omega)$ extracts the numerical quantity we care about.

Examples:

• Coin toss with payout — if heads you win $1, tails you lose$1. Then $X(\text{H}) = 1$, $X(\text{T}) = -1$.
• Sum of two dice — $X((i, j)) = i + j$.
• Tomorrow's temperature in Celsius — $X$ is the reading itself (identity map).

RVs let us talk about distributions rather than raw sample spaces. Instead of computing $P(\{\omega: X(\omega) \le 3\})$ from scratch every time, we characterise the behaviour of $X$ through its distribution functions.

RVs are discrete if $X$ takes values in a finite or countably infinite set (e.g., $\{0, 1\}$, $\mathbb{N}$). They are continuous if $X$ can take any value in an interval or $\mathbb{R}$.

PMF, PDF, CDF#

Three functions completely describe any random variable. Each answers a slightly different question.

Probability Mass Function (PMF) — discrete only#

For a discrete RV $X$, the PMFProbability Mass Function is

Requirements: $p_X(x) \ge 0$ for all $x$, and $\sum_x p_X(x) = 1$.

Worked example — biased die. A loaded die has PMF: $p(1) = p(2) = p(3) = 0.1$, $p(4) = p(5) = 0.15$, $p(6) = 0.4$. Check: $3(0.1) + 2(0.15) + 0.4 = 0.3 + 0.3 + 0.4 = 1.0$. Probability of rolling $\ge 5$ is $p(5) + p(6) = 0.15 + 0.4 = 0.55$.

Probability Density Function (PDF) — continuous only#

For a continuous RV $X$, the PDFProbability Density Function $f_X(x)$ is a non-negative function that integrates to 1:

Probability of an interval is the area under the curve:

Critical: $f_X(x)$ is not a probability. It is probability density — probability per unit of $x$. For continuous RVs, $P(X = c) = 0$ for every single point $c$, even if $f_X(c) > 0$. Probability is area, not height.

Cumulative Distribution Function (CDF) — always exists#

The CDFCumulative Distribution Function is defined for any RV, discrete or continuous:

Properties:

• $F_X$ is non-decreasing.
• $\lim_{x \to -\infty} F_X(x) = 0$, $\lim_{x \to +\infty} F_X(x) = 1$.
• For continuous RVs, $F_X$ is smooth and $F_X'(x) = f_X(x)$ (where $F_X$ is differentiable).
• For discrete RVs, $F_X$ is a step function with jumps at each possible value; the jump size equals the PMF value.

CDF computes interval probabilities: $P(a < X \le b) = F_X(b) - F_X(a)$.

Worked example — Uniform on $[0,1]$:

Then $P(0.3 \le X \le 0.7) = F(0.7) - F(0.3) = 0.7 - 0.3 = 0.4$.

Bernoulli and Binomial#

Bernoulli distribution#

A single trial with two outcomes: success (1) with probability $p$, failure (0) with probability $1-p$.

Notation: $X \sim \text{Bernoulli}(p)$. The PMF is $p_X(x) = p^x (1-p)^{1-x}$ for $x \in \{0, 1\}$.

Example: A classifier makes the correct prediction with probability 0.9. Each prediction is a $\text{Bernoulli}(0.9)$ trial.

Binomial distribution#

$n$ independent Bernoulli$(p)$ trials; $X$ counts the number of successes:

Notation: $X \sim \text{Binomial}(n, p)$. The binomial coefficient $\binom{n}{k} = \frac{n!}{k!(n-k)!}$ counts the number of ways to choose which $k$ of the $n$ trials are successes.

Worked example. A factory produces chips; each chip is defective independently with probability $p = 0.05$. In a batch of $n = 20$ chips, what is the probability of exactly 2 defectives?

Binomial as sum of Bernoullis: If $Y_1, \ldots, Y_n \stackrel{\text{i.i.d.}}{\sim} \text{Bernoulli}(p)$, then $X = \sum_{i=1}^n Y_i \sim \text{Binomial}(n, p)$. This is a key insight: complex distributions are often built from simpler ones by summation.

Uniform and Exponential#

Uniform distribution (continuous)#

All values in an interval are equally likely. If $X \sim \text{Uniform}(a, b)$:

The Uniform is the continuous analogue of equally likely discrete outcomes. It models "complete ignorance" over a bounded range — e.g., a bus arriving uniformly at random between 10:00 and 10:15.

Exponential distribution#

The Exponential models waiting times between independent events (e.g., time until the next radioactive decay, next customer arrival). If $X \sim \text{Exponential}(\lambda)$ with rate $\lambda > 0$:

Memoryless property (mention): $P(X > s + t \mid X > s) = P(X > t)$. The distribution of remaining wait time does not depend on how long you have already waited — unique to the Exponential among continuous distributions. This makes it a natural model for systems where events occur "at random" with no aging.

Worked example. Server requests arrive at rate $\lambda = 2$ per second. What is $P(\text{wait} \le 0.5\,\text{sec})$?

Choosing a Model#

Every distribution tells a story. Choosing the right model means matching the data-generating process to the distribution's story:

| Story | Distribution | |---|---| | Single yes/no trial with probability $p$ | Bernoulli$(p)$ | | Count of successes in $n$ independent trials | Binomial$(n, p)$ | | Random value in a bounded interval, all values equally plausible | Uniform$(a, b)$ | | Waiting time with constant hazard (no memory) | Exponential$(\lambda)$ |

Key questions to ask:

1. Is the quantity discrete (countable) or continuous (measured)?
2. Is there a natural upper bound (Binomial has $n$; Uniform has $a, b$)?
3. Are trials independent? (If not, Binomial is wrong.)
4. Does the process reset (memoryless)? (If yes, Exponential may fit.)

A model is never "true" — it is a useful approximation. The art is choosing one that captures the essential structure while remaining tractable.

Knowledge check#

Browser lab#

Sample from named distributions and compare the histogram to the true PMF/PDF curve.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(42)
n = 10000

fig, axes = plt.subplots(2, 2, figsize=(10, 8))

# Binomial
binom = rng.binomial(n=20, p=0.3, size=n)
ks = np.arange(0, 21)
pmf_true = np.array([np.math.comb(20, k) * 0.3**k * 0.7**(20 - k) for k in ks])
axes[0, 0].bar(ks, pmf_true, alpha=0.5, label="True PMF")
axes[0, 0].hist(binom, bins=np.arange(-0.5, 21.5, 1), density=True, alpha=0.7, label="Empirical")
axes[0, 0].set_title("Binomial(20, 0.3)")
axes[0, 0].legend()

# Uniform
unif = rng.uniform(2, 6, size=n)
x_unif = np.linspace(1, 7, 200)
pdf_unif = np.where((x_unif >= 2) & (x_unif <= 6), 0.25, 0)
axes[0, 1].plot(x_unif, pdf_unif, "tab:blue", label="True PDF")
axes[0, 1].hist(unif, bins=40, density=True, alpha=0.7, label="Empirical")
axes[0, 1].set_title("Uniform(2, 6)")
axes[0, 1].legend()

# Exponential
exp_samples = rng.exponential(scale=1/1.5, size=n)
x_exp = np.linspace(0, 5, 200)
pdf_exp = 1.5 * np.exp(-1.5 * x_exp)
axes[1, 0].plot(x_exp, pdf_exp, "tab:blue", label="True PDF")
axes[1, 0].hist(exp_samples, bins=60, density=True, alpha=0.7, label="Empirical")
axes[1, 0].set_title("Exponential(λ=1.5)")
axes[1, 0].legend()

# Empirical CDF vs true CDF for Exponential
sorted_exp = np.sort(exp_samples)
empirical_cdf = np.arange(1, n + 1) / n
true_cdf = 1 - np.exp(-1.5 * sorted_exp)
axes[1, 1].step(sorted_exp, empirical_cdf, label="Empirical CDF")
axes[1, 1].plot(sorted_exp, true_cdf, label="True CDF")
axes[1, 1].set_title("CDF: Exponential(λ=1.5)")
axes[1, 1].legend()

plt.tight_layout()
plt.show()

print(f"Samples: {n}")
print(f"Binomial mean (true=6.0): {binom.mean():.3f}")
print(f"Uniform mean (true=4.0): {unif.mean():.3f}")
print(f"Exponential mean (true=0.667): {exp_samples.mean():.3f}")