Week 1: Probability Spaces and Events

Why Probability for Machine Learning#

Machine learning operates on uncertainty. A classifier outputs a confidence score, not a deterministic label. A reinforcement learning agent acts in stochastic environments where the same action does not always produce the same result. A generative model learns a distribution over data, not a single point estimate.

Probability gives us the language to describe this uncertainty precisely. It tells us how to update beliefs when we see data (Week 5), how to quantify the spread of an estimator (Week 10), and how to measure the difference between distributions (Week 11). The courses on Reinforcement Learning and Generative Models speak this language fluently — this course builds your vocabulary from the ground up.

The approach we take is axiomatic: we agree on three simple rules that any coherent assignment of probabilities must obey. From those rules, everything else follows.

Sample Spaces and Events#

Every probabilistic model starts with a sample space $\Omega$ — the set of all possible outcomes of an experiment.

Examples:

• Toss a fair coin: $\Omega = \{\text{H}, \text{T}\}$.
• Roll a six-sided die: $\Omega = \{1, 2, 3, 4, 5, 6\}$.
• Measure tomorrow's temperature (in °C): $\Omega = \mathbb{R}$ (or an interval).
• Two coin tosses: $\Omega = \{\text{HH}, \text{HT}, \text{TH}, \text{TT}\}$.

An event is a subset of $\Omega$ — a collection of outcomes about which we can ask "what is the probability?".

Example — die roll. Let $\Omega = \{1, 2, 3, 4, 5, 6\}$. The event "roll an even number" is $E = \{2, 4, 6\}$. The event "roll at least 5" is $F = \{5, 6\}$.

Set operations on events (since events are sets, set language applies):

• Union $A \cup B$: the event that $A$ or $B$ occurs (or both).
• Intersection $A \cap B$: the event that $A$ and $B$ both occur.
• Complement $A^c = \Omega \setminus A$: the event that $A$ does not occur.
• Disjoint (mutually exclusive) events: $A \cap B = \varnothing$, meaning they cannot both happen on the same realisation.

For the die example: $E \cap F = \{6\}$ (even and at least 5), $E \cup F = \{2, 4, 5, 6\}$.

Kolmogorov Axioms#

A probability measure $P$ assigns a number in $[0, 1]$ to each event. Andrey Kolmogorov (1933) gave three axioms that any valid $P$ must satisfy:

1. Non-negativity. For every event $A$, $P(A) \ge 0$.
2. Normalisation. $P(\Omega) = 1$ — some outcome must occur.
3. Countable additivity (finite version). If $A_1, A_2, \ldots, A_k$ are pairwise disjoint, then

For most of this course we use the finite version of axiom 3: probabilities of mutually exclusive events simply add. The infinite (countable) version handles sequences, but the intuition is identical.

What the axioms give us — useful consequences:

• $P(\varnothing) = 0$ (the impossible event).
• Complement rule: $P(A^c) = 1 - P(A)$.
• Monotonicity: If $A \subseteq B$, then $P(A) \le P(B)$.
• Inclusion-exclusion (two events):

The inclusion-exclusion formula corrects for double-counting: adding $P(A) + P(B)$ counts the intersection twice, so we subtract it once.

Worked example. In a class of 100 students, 60 study calculus ($A$), 45 study linear algebra ($B$), and 20 study both. What is the probability a randomly chosen student studies at least one of the two subjects?

So 85% study at least one. The complement rule gives $P(\text{neither}) = 1 - 0.85 = 0.15$.

Finite Spaces and Counting#

When $\Omega$ is finite and every outcome is equally likely, probability reduces to counting:

where $|\cdot|$ denotes cardinality (number of elements). This is the "classical" definition of probability — but it only works when the equiprobable assumption is justified (fair coins, fair dice, well-shuffled cards, random sampling).

Worked example — two dice. Roll two fair six-sided dice (36 equally likely ordered pairs). What is the probability that the sum is 7?

The outcomes that sum to 7 are $\{(1,6), (2,5), (3,4), (4,3), (5,2), (6,1)\}$ — six outcomes. So

Inclusion-exclusion with counting (two events):

For two events $A$ and $B$ in a finite equiprobable space:

Dividing through by $|\Omega|$ recovers the probability inclusion-exclusion formula above. The three-event version is:

The pattern: add singles, subtract pairs, add triples — alternating signs so every outcome in the union is counted exactly once.

Discrete vs Continuous Setup#

So far we have thought of $\Omega$ as a finite or countably infinite set (e.g., $\mathbb{N}$). In that case we can list the outcomes and assign a probability $p(\omega)$ to each $\omega \in \Omega$, with $\sum_{\omega} p(\omega) = 1$. For any event $A$, $P(A) = \sum_{\omega \in A} p(\omega)$.

But many quantities of interest — temperature, height, the weight of a neural network — are continuous. The sample space is an interval or all of $\mathbb{R}$. Here the list-and-sum approach breaks down: any single real number has probability zero (think of picking a point on a line — there are uncountably many of them, so each gets measure zero).

For continuous spaces, probability is assigned via a density function $f(x) \ge 0$ that integrates to 1 over the space:

This is the subject of Week 2, where we introduce random variables, PMFs, PDFs, and CDFs systematically. For now, the key distinction is:

| Discrete | Continuous | |---|---| | $\Omega$ is finite or countable | $\Omega$ is an interval or $\mathbb{R}$ | | $P(A) = \sum_{\omega \in A} p(\omega)$ | $P(A) = \int_A f(x)\, dx$ | | Single outcomes can have positive probability | Single outcomes have probability zero | | Motivating examples: coins, dice, cards | Motivating examples: measurements, sensor readings |

The axiomatic framework (Kolmogorov's three rules) works in both settings. The only difference is whether we sum or integrate to accumulate probability over an event.

Warning — Do not conflate the two. In a continuous setting, $P(X = 3.7)$ is 0, even though the density $f(3.7)$ may be positive. Density is not probability — it is probability per unit. Week 2 makes this crisp.

Knowledge check#

Browser lab#

Simulate die rolls and compare empirical frequencies to the axiom-based probabilities of events.

import numpy as np
import matplotlib.pyplot as plt

def roll_die(n):
    return np.random.randint(1, 7, size=n)

n_rolls = 6000
rolls = roll_die(n_rolls)

events = {
    "Even (2,4,6)": (rolls % 2 == 0),
    "≥ 5 (5,6)": (rolls >= 5),
    "Prime (2,3,5)": np.isin(rolls, [2, 3, 5]),
    "Divisible by 3 (3,6)": (rolls % 3 == 0),
}

true_probs = {"Even (2,4,6)": 0.5, "≥ 5 (5,6)": 1/3,
              "Prime (2,3,5)": 0.5, "Divisible by 3 (3,6)": 1/3}

fig, axes = plt.subplots(2, 2, figsize=(8, 7))
for ax, (name, mask) in zip(axes.flat, events.items()):
    empirical = mask.sum() / n_rolls
    ax.bar(["Axiom (true)", "Empirical"], [true_probs[name], empirical],
           color=["tab:blue", "tab:orange"])
    ax.set_ylim(0, 1)
    ax.set_title(name)
    ax.set_ylabel("Probability")

fig.suptitle(f"Empirical vs True Probabilities ({n_rolls} rolls)", fontsize=13)
plt.tight_layout()
plt.show()

print("Empirical frequencies:")
for name, mask in events.items():
    print(f"  {name}: {mask.sum() / n_rolls:.4f}  (true: {true_probs[name]:.4f})")