Week 7: Orthogonality and Projections

Orthogonal Vectors and Subspaces#

Week 1 defined the dot product of two vectors $u, v \in \mathbb{R}^n$ as

Two vectors are orthogonal when their dot product is zero:

Geometrically, orthogonal vectors meet at a right angle. Algebraically, the test is a single number. The zero vector is orthogonal to every vector, because $0^\top v = 0$ always.

Orthogonal subspaces. Two subspaces $V, W \subseteq \mathbb{R}^n$ are orthogonal when every vector in one is orthogonal to every vector in the other:

It is enough to check bases: if every basis vector of $V$ is orthogonal to every basis vector of $W$, the whole subspaces are orthogonal (by bilinearity of the dot product).

Orthogonal complements and the four subspaces. Week 6 introduced the four fundamental subspaces of an $m \times n$ matrix $A$. Orthogonality locks them into complementary pairs:

• In $\mathbb{R}^n$: the row space $C(A^\top)$ and the null space $N(A)$ are orthogonal complements. Every row of $A$ is orthogonal to every $x$ with $Ax = 0$ (because $Ax = 0$ means each row dotted with $x$ is zero). Dimensions add: $\dim C(A^\top) + \dim N(A) = r + (n - r) = n$, so together they fill all of $\mathbb{R}^n$.
• In $\mathbb{R}^m$: the column space $C(A)$ and the left null space $N(A^\top)$ are orthogonal complements. If $A^\top y = 0$, then $y$ is orthogonal to every column of $A$. Dimensions add: $r + (m - r) = m$.

We write $N(A) = C(A^\top)^\perp$ and $N(A^\top) = C(A)^\perp$. That is the structural fact Week 6 previewed: the kernel of $A$ is exactly the orthogonal complement of the row space, and the left null space is the orthogonal complement of the column space.

Why this matters for projections. Any vector $b \in \mathbb{R}^m$ decomposes uniquely as

with $p \perp e$. The piece $p$ is the orthogonal projection of $b$ onto the column space — the closest point of $C(A)$ to $b$. The rest of this week builds the formulas that produce $p$.

Projection onto a Line#

Start with the simplest subspace: a line through the origin spanned by a nonzero vector $a \in \mathbb{R}^n$. Given a vector $b$, we want the point $p$ on that line closest to $b$. Write

for some scalar $\hat{x}$. The error (residual) is $e = b - p = b - \hat{x}\, a$. For $p$ to be the orthogonal projection, the error must be orthogonal to the line — that is, orthogonal to $a$:

Expand and solve for $\hat{x}$:

Therefore the projection is

Geometric reading. The scalar $\hat{x}$ is the coordinate of $b$ along the unit direction of $a$, scaled by $\|a\|$. The formula never needs an explicit unit vector: the factor $1/(a^\top a)$ absorbs the length of $a$.

Worked example. Project $b = \begin{pmatrix} 4 \\ 2 \end{pmatrix}$ onto the line spanned by $a = \begin{pmatrix} 1 \\ 1 \end{pmatrix}$.

So

The error is $e = b - p = \begin{pmatrix} 1 \\ -1 \end{pmatrix}$, and $a^\top e = 1 - 1 = 0$ — orthogonal, as required. Notice $\|e\|^2 = 2$ is the squared distance from $b$ to the line; any other point on the line would be farther.

Special cases.

• If $b$ already lies on the line, then $b = ca$ for some $c$, and $\hat{x} = c$, so $p = b$.
• If $b \perp a$, then $a^\top b = 0$ and $p = 0$ — the nearest point on the line is the origin.

Projection onto a Subspace#

Now replace the line $\operatorname{span}\{a\}$ by a higher-dimensional subspace. Let the columns of an $m \times n$ matrix $A$ form a basis for the subspace we project onto (so $A$ has full column rank $n$, and the subspace is $C(A) \subseteq \mathbb{R}^m$). Any point in the subspace has the form $A x$ for a unique $x \in \mathbb{R}^n$. We seek $\hat{x}$ so that

is the orthogonal projection of $b$ onto $C(A)$. Again the error $e = b - A\hat{x}$ must be orthogonal to every vector in the subspace — in particular, orthogonal to every column of $A$:

Rearrange:

These are the normal equations. Because $A$ has full column rank, $A^\top A$ is invertible (it is $n \times n$, symmetric, and positive definite: $x^\top A^\top A x = \|Ax\|^2 = 0$ forces $Ax = 0$ and hence $x = 0$). Solve:

Reduction to the line formula. When $A$ is a single column $a$, the matrix $A^\top A$ is the scalar $a^\top a$, and $(A^\top A)^{-1} A^\top b$ becomes $(a^\top b)/(a^\top a)$ — exactly the line formula from the previous section.

Worked example. Let

The two columns of $A$ are independent, so they span a plane in $\mathbb{R}^3$. Compute

Then

In this case $p = b$: the target already lies in $C(A)$. The residual is zero, and the normal equations hold trivially.

If $b$ is not in $C(A)$. The same formulas still produce the closest point $p \in C(A)$. The residual $e = b - p$ lands in $N(A^\top)$ — the left null space from Week 6 — so $A^\top e = 0$ by construction. Week 8 will use exactly this setup for least squares: solve $A\hat{x} \approx b$ by minimizing $\|b - Ax\|$.

The Projection Matrix#

The subspace formula $p = A(A^\top A)^{-1} A^\top b$ is linear in $b$. The matrix that implements it is the projection matrix

Then $p = P b$ for every $b$. Once $P$ is built from $A$, projecting any number of vectors is a single matrix–vector multiply.

Two algebraic properties characterize orthogonal projections:

1. Idempotence: $P^2 = P$.
   Projecting twice does nothing new: the first application lands in $C(A)$, and vectors already in $C(A)$ are fixed by $P$. Explicitly,

   $$P^2 = A(A^\top A)^{-1} A^\top \cdot A(A^\top A)^{-1} A^\top = A(A^\top A)^{-1} A^\top = P,$$

   because $A^\top A (A^\top A)^{-1} = I$ cancels in the middle.

2. Symmetry: $P^\top = P$.
   Orthogonal projection is a symmetric linear map. Taking the transpose of the formula,

   $$P^\top = \bigl(A(A^\top A)^{-1} A^\top\bigr)^\top = A\bigl((A^\top A)^{-1}\bigr)^\top A^\top = A(A^\top A)^{-1} A^\top = P,$$

   using that $A^\top A$ (and its inverse) is symmetric.

Conversely, any matrix that is both symmetric and idempotent is the orthogonal projection onto its column space.

Geometry of the properties.

• $P^2 = P$ says the map is a projection (not necessarily orthogonal): applying it again cannot move the result.
• $P^\top = P$ forces the projection to be orthogonal — the residual $b - Pb$ is orthogonal to the subspace. Without symmetry you can have oblique projections that still satisfy $P^2 = P$ but do not minimize distance.

Eigenvalues. Because $P^2 = P$, every eigenvalue $\lambda$ satisfies $\lambda^2 = \lambda$, so $\lambda \in \{0, 1\}$. The eigenspace for $\lambda = 1$ is $C(P) = C(A)$ (the subspace we project onto); the eigenspace for $\lambda = 0$ is the orthogonal complement $N(P) = N(A^\top)$. Trace equals rank: $\operatorname{tr}(P) = \operatorname{rank}(P) = n$ when $A$ is $m \times n$ with full column rank.

Line case again. For a single nonzero column $a$,

the familiar outer-product formula. Check: $P a = a$ and $P v = 0$ whenever $v \perp a$.

Knowledge check#

Browser lab#

Build a projection matrix onto a 2D subspace of $\mathbb{R}^3$ and verify $P^2 = P$ (idempotence) and $P^\top = P$ (symmetry). The columns of $A$ span the subspace; $p = P b$ is the orthogonal projection of $b$ onto that plane.

import numpy as np

A = np.array([[1., 0.],
              [1., 1.],
              [0., 1.]])
b = np.array([2., 3., 1.])

P = A @ np.linalg.inv(A.T @ A) @ A.T
p = P @ b

print("Projection matrix P =\n", P)
print("Projected vector p =", p)
print("P @ P ≈ P:", np.allclose(P @ P, P))
print("P.T ≈ P:", np.allclose(P.T, P))
print("residual A.T @ (b - p) ≈ 0:", np.allclose(A.T @ (b - p), 0))