Week 4: Conditioning and Independence — Prob & Stats

Conditional Probability#

Probability describes uncertainty given what we know. When we learn that event $B$ has occurred, probabilities of other events should update. The conditional probability of $A$ given $B$ is:

P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \quad \text{provided } P(B) > 0.

Think of $B$ as the "new universe": we restrict attention to outcomes in $B$ , then ask what fraction of those also satisfy $A$ .

Worked example — cards. Draw one card from a standard 52-card deck. Let $A$ = "card is an Ace", $B$ = "card is a spade". Then $P(A) = 4/52 = 1/13$ , but

P(A \mid B) = \frac{P(A \cap B)}{P(B)} = \frac{1/52}{13/52} = \frac{1}{13}.

$P(A \mid B) = P(A)$ — interesting: knowing the suit doesn't change the probability of an Ace. This is independence (defined below).

Multiplication rule (rearranging the definition):

P(A \cap B) = P(A \mid B) \, P(B) = P(B \mid A) \, P(A).

This is how we factor joint probabilities into a chain of conditionals.

Worked example — two draws without replacement. Draw two cards from a deck. What is the probability both are Aces? Let $A_1$ = first is Ace, $A_2$ = second is Ace.

P(A_1 \cap A_2) = P(A_2 \mid A_1) \, P(A_1) = \frac{3}{51} \cdot \frac{4}{52} = \frac{12}{2652} \approx 0.0045.

After drawing the first Ace, only 3 Aces remain among the 51 remaining cards — conditioning captures this.

Total Probability#

The law of total probability partitions the sample space and weights conditional probabilities:

If $B_1, B_2, \ldots, B_k$ form a partition of $\Omega$ (mutually exclusive and exhaustive), then

P(A) = \sum_{i=1}^k P(A \mid B_i) \, P(B_i).

This is the weighted average of conditional probabilities, where the weights are the probabilities of the conditioning events.

Worked example — factory lines. A factory has three production lines. Line 1 produces 50% of units with a 1% defect rate. Line 2 produces 30% at 2%. Line 3 produces 20% at 5%. If we pick a unit at random, what is $P(\text{defective})$ ?

P(D) = 0.01(0.50) + 0.02(0.30) + 0.05(0.20) = 0.005 + 0.006 + 0.010 = 0.021.

So 2.1% of all units are defective. The law of total probability will combine with Bayes' rule in Week 5 for the reverse inference: given a defective unit, which line did it come from?

Independence#

Events $A$ and $B$ are independent if learning about one gives no information about the other:

P(A \cap B) = P(A) \, P(B).

Equivalently: $P(A \mid B) = P(A)$ (when $P(B) > 0$ ), and $P(B \mid A) = P(B)$ (when $P(A) > 0$ ).

Worked example — coin and die. Toss a fair coin and roll a fair die. Let $A$ = "coin is Heads", $B$ = "die shows 6". Since the outcomes occur independently:

P(A \cap B) = \frac{1}{2} \cdot \frac{1}{6} = \frac{1}{12}.

Pairwise vs mutual independence. For three events, pairwise independence ( $A \perp B$ , $B \perp C$ , $A \perp C$ ) does not guarantee mutual independence. Here is a classic counterexample:

Toss two fair coins. Let $A$ = first coin heads, $B$ = second coin heads, $C$ = both coins match (both heads or both tails). Then $P(A) = P(B) = P(C) = 0.5$ . Check pairwise:

$P(A \cap B) = 1/4 = P(A)P(B)$ — independent.
$P(A \cap C) = P(\text{HH}) = 1/4 = P(A)P(C)$ — independent.
$P(B \cap C) = 1/4 = P(B)P(C)$ — independent.

But $P(A \cap B \cap C) = P(\text{HH}) = 1/4$ , while $P(A)P(B)P(C) = 1/8$ . The events are pairwise independent but not mutually independent. Mutual independence requires $P(A \cap B \cap C) = P(A)P(B)P(C)$ and the pairwise conditions.

Chain Rule#

The chain rule (also called the product rule) factors a joint probability over $n$ variables into a product of conditionals:

P(X_1, X_2, \ldots, X_n) = P(X_1) \cdot P(X_2 \mid X_1) \cdot P(X_3 \mid X_1, X_2) \cdots P(X_n \mid X_1, \ldots, X_{n-1}).

Each step conditions on all previous variables. This is always valid — no independence assumptions are needed. The order of variables can be chosen arbitrarily; different orderings give different but equivalent factorisations.

Worked example — three draws. Draw three cards without replacement from a standard deck. Probability all three are spades:

P(S_1, S_2, S_3) = P(S_1) \cdot P(S_2 \mid S_1) \cdot P(S_3 \mid S_1, S_2) = \frac{13}{52} \cdot \frac{12}{51} \cdot \frac{11}{50} = \frac{1716}{132600} \approx 0.0129.

The chain rule is the backbone of autoregressive models in ML (e.g., language models predict token $t$ given tokens $1, \ldots, t-1$ ) and of Bayesian networks (Week 13), where conditional independence prunes the conditioning sets.

Conditional Independence#

$X$ and $Y$ are conditionally independent given $Z$ (written $X \perp\!\!\!\perp Y \mid Z$ ) if, once you know $Z$ , learning $Y$ gives no additional information about $X$ :

P(X, Y \mid Z) = P(X \mid Z) \, P(Y \mid Z).

Equivalently: $P(X \mid Y, Z) = P(X \mid Z)$ .

Conditional independence is different from (marginal) independence. Variables can be:

Marginally dependent but conditionally independent.
Marginally independent but conditionally dependent.

Story — Naive Bayes preview. In document classification, the words in a document are not independent (seeing "gradient" increases the probability of seeing "descent"). But if we know the document topic (say, "machine learning"), the appearance of individual words becomes approximately independent: $P(\text{"gradient"}, \text{"descent"} \mid \text{topic}=\text{ML}) \approx P(\text{"gradient"} \mid \text{ML}) \cdot P(\text{"descent"} \mid \text{ML})$ .

This is the naive Bayes assumption: features are conditionally independent given the class label. It is "naive" because it is rarely exactly true, yet it works surprisingly well in practice because the probability rankings of classes are often correct even when the absolute probabilities are off.

Why this matters for graphical models (Week 13): conditional independence statements are compactly encoded by graph structure. A missing edge between two nodes means they are conditionally independent given their neighbours.

Exercise · Fill in the blank

From a two-way table: P(A ∩ B)=0.15, P(B)=0.3. Compute P(A | B) as a decimal.

Knowledge check#

Question 1 of 4

The definition of conditional probability P(A | B) is:

P(A ∩ B) / P(B)

P(A ∩ B) / P(A)

P(A) P(B)

P(A) / P(B)

Browser lab#

Simulate pairs of variables and compute empirical conditional probabilities, comparing to the true conditional table.

python · runs in browser

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(42)
n = 10000

p_x = np.array([0.3, 0.7])
p_y_given_x = np.array([[0.2, 0.5, 0.3],
                         [0.6, 0.3, 0.1]])

xs = rng.choice([0, 1], size=n, p=p_x)
ys = np.array([rng.choice([0, 1, 2], p=p_y_given_x[x]) for x in xs])

joint_emp = np.zeros((2, 3))
for xv in range(2):
    for yv in range(3):
        joint_emp[xv, yv] = np.mean((xs == xv) & (ys == yv))

p_y_given_x_emp = joint_emp / joint_emp.sum(axis=1, keepdims=True)

print("  True P(Y | X):")
print(p_y_given_x)
print("  Empirical P(Y | X):")
print(np.round(p_y_given_x_emp, 4))

print(f"\n  P(Y=0): true={np.sum(p_y_given_x * p_x[:, None], axis=0)[0]:.3f}")
print(f"           emp={np.mean(ys == 0):.3f}")

fig, axes = plt.subplots(1, 2, figsize=(10, 4))
for xv, ax, label in zip([0, 1], axes, ["X=0", "X=1"]):
    mask = xs == xv
    ax.hist(ys[mask], bins=np.arange(-0.5, 3.5, 1), density=True,
            alpha=0.5, label="P(Y | X) empirical")
    ax.bar([0, 1, 2], p_y_given_x[xv], alpha=0.4, label="P(Y | X) true")
    ax.set_title(f"Conditional distribution of Y given {label}")
    ax.set_xlabel("Y"); ax.set_ylabel("Probability")
    ax.legend()
plt.tight_layout()
plt.show()

← Previous

Week 3: Expectation, Variance, and Covariance

Week 5: Bayes' Rule and Generative Classification