Week 12: Symmetric and Positive Definite Matrices

The Spectral Theorem#

A real matrix $A$ is symmetric when $A = A^T$ — every entry equals its transpose partner, so $a_{ij} = a_{ji}$. Symmetric matrices appear everywhere: covariance matrices, Hessian matrices of smooth functions, graph Laplacians, and Gram matrices of the form $B^TB$. This week is about what symmetry buys you algebraically and geometrically.

The spectral theorem is the central fact:

Every real symmetric $n \times n$ matrix $A$ has $n$ real eigenvalues $\lambda_1,\ldots,\lambda_n$ (counting multiplicity) and an orthonormal basis of eigenvectors $q_1,\ldots,q_n$. In matrix form

$$A = Q\Lambda Q^T,$$

where $\Lambda = \operatorname{diag}(\lambda_1,\ldots,\lambda_n)$ and $Q$ is an orthogonal matrix ($Q^TQ = I$, so $Q^{-1} = Q^T$) whose columns are the eigenvectors.

Compare with ordinary diagonalization $A = S\Lambda S^{-1}$ from Week 10. Symmetry upgrades two things at once:

1. All eigenvalues are real — no complex conjugate pairs, no hidden oscillation from imaginary parts.
2. The eigenvector matrix can be chosen orthogonal — $S^{-1}$ is simply $S^T$, and the change of coordinates is a pure rotation/reflection (an isometry of $\mathbb{R}^n$).

Why eigenvectors for distinct eigenvalues are orthogonal. Suppose $Aq_1 = \lambda_1 q_1$, $Aq_2 = \lambda_2 q_2$, with $\lambda_1 \neq \lambda_2$ and $A = A^T$. Take the inner product:

So $(\lambda_1 - \lambda_2)\, q_2^T q_1 = 0$. Because $\lambda_1 \neq \lambda_2$, we must have $q_2^T q_1 = 0$: the eigenvectors are orthogonal. When eigenvalues repeat, you can still orthonormalize within each eigenspace (Gram–Schmidt from Week 7) and keep different eigenspaces orthogonal. The result is always an orthonormal basis of $\mathbb{R}^n$.

Worked check. Let

Trace $4$, determinant $3$, so eigenvalues solve $\lambda^2 - 4\lambda + 3 = 0$: $\lambda_1 = 3$, $\lambda_2 = 1$. For $\lambda = 3$:

For $\lambda = 1$:

Indeed $q_1^T q_2 = 0$, and

(You can verify $Q^TQ = I$ and $Q\Lambda Q^T = A$ by direct multiplication.)

Quadratic Forms#

A quadratic form is a scalar function of a vector built from a symmetric matrix:

In components for $n = 2$ with $A = \begin{pmatrix} a & b \\ b & c \end{pmatrix}$ this is

(The off-diagonal $b$ contributes twice because $a_{12}$ and $a_{21}$ both equal $b$.) Quadratic forms measure energy, squared length after a linear map ($\|Bx\|^2 = x^T(B^TB)x$), and second-order Taylor terms of a smooth function near a critical point.

Diagonalization simplifies the form. Write $A = Q\Lambda Q^T$ and change variables to the eigen-coordinates $y = Q^T x$ (so $x = Qy$). Then

The cross terms vanish. In the eigenvector basis, the quadratic form is a pure weighted sum of squares: $\lambda_1 y_1^2 + \cdots + \lambda_n y_n^2$. That is why the spectral theorem is the engine of everything that follows — positive definiteness, ellipses, and (next week) the SVD all reduce to the signs and sizes of the $\lambda_i$.

Example. For the matrix above with $\lambda = 3,1$,

in the $Q$-coordinates. Along the first eigen-direction the form grows thrice as fast as along the second.

Positive Definite Matrices#

A symmetric matrix $A$ is positive definite (often written $A \succ 0$) when

If the inequality is $\ge 0$ instead of $> 0$, $A$ is positive semidefinite. Negative definite means $x^TAx < 0$ for all $x \neq 0$ (equivalently, $-A$ is positive definite).

Positive definite matrices are the multivariable analogue of a positive number: they define genuine energies, strict convexity of $x^TAx/2$, and unique minima of associated least-squares problems. Three classical tests decide positive definiteness — all equivalent for real symmetric $A$.

Test 1 — Eigenvalues#

From the spectral form $x^TAx = \sum \lambda_i y_i^2$ with $y = Q^Tx$ and $\|y\| = \|x\|$ (because $Q$ is orthogonal), we get:

• $A$ is positive definite $\Longleftrightarrow$ every eigenvalue $\lambda_i > 0$.
• $A$ is positive semidefinite $\Longleftrightarrow$ every $\lambda_i \ge 0$.
• $A$ is indefinite if some $\lambda_i > 0$ and some $\lambda_j < 0$ (the form takes both positive and negative values).

This is usually the conceptual test: stretch factors in every eigen-direction must be positive.

Test 2 — Pivots#

Eliminate $A$ by Gaussian elimination without row exchanges (or more carefully: with pivots that stay on the diagonal when possible). The successive pivots $d_1, d_2, \ldots, d_n$ that appear on the diagonal of the upper-triangular factor are all positive if and only if $A$ is positive definite. Equivalently, the LDL$^T$ factorization $A = LDL^T$ has a positive diagonal $D$.

Test 3 — Leading principal minors (determinants)#

Let $A_k$ be the upper-left $k \times k$ principal submatrix of $A$ (the first $k$ rows and columns). The leading principal minors are $\det(A_1),\ldots,\det(A_n)$. Sylvester's criterion:

For a $2 \times 2$ matrix $\begin{pmatrix} a & b \\ b & c \end{pmatrix}$ this is simply $a > 0$ and $ac - b^2 > 0$.

Worked example. Take again

• Eigenvalues: $\lambda = 3$ and $\lambda = 1$, both positive $\Rightarrow$ positive definite.
• Pivots: first pivot $2 > 0$; after eliminating, the Schur complement is $2 - (-1)^2/2 = 3/2 > 0$. Both pivots positive.
• Leading determinants: $\det(A_1) = 2 > 0$ and $\det(A) = 4 - 1 = 3 > 0$.

All three tests agree. Contrast with $B = \begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix}$: eigenvalues $3$ and $-1$, so $B$ is indefinite — $\det(B) = -3 < 0$ already fails the leading-determinant test.

Why three tests? Eigenvalues are conceptual. Pivots are computational during elimination. Leading determinants are a quick hand check for small $n$. Use whichever is handy; they always give the same yes/no answer for symmetric $A$.

Geometric Picture#

Fix a positive definite $A$ and a positive constant $c$. The level set

is a closed bounded surface around the origin. In the eigen-coordinates $y = Q^T x$ the equation becomes

or equivalently

That is the standard equation of an ellipse when $n = 2$, and an ellipsoid when $n \ge 3$. The geometry is completely determined by the spectral data:

| Feature | Determined by | |---|---| | Orientation of the axes | Eigenvectors (columns of $Q$) | | Axis half-lengths for level $c$ | $\sqrt{c / \lambda_i}$ along eigenvector $q_i$ | | “Fatness” vs “thinness” | Ratio of eigenvalues (condition number of $A$) |

Large $\lambda_i$ means a short axis: the quadratic form grows quickly in that direction, so the level set $x^TAx = c$ is reached sooner. Small $\lambda_i$ means a long axis. When one eigenvalue is much larger than another, the ellipse is elongated along the small-eigenvalue direction.

If $A$ is not positive definite. Zero eigenvalues open the level set into a cylinder (unbounded in the null direction). Opposite-sign eigenvalues produce hyperbolas — the form is indefinite and the “energy” $x^TAx$ can be positive, zero, or negative depending on direction.

Preview of Week 13. For any matrix $B$ (not necessarily square), the Gram matrix $A = B^TB$ is always symmetric and positive semidefinite. Its eigenvalues are the squared singular values of $B$, and its eigenvectors are the right singular vectors. The spectral theorem for $A$ is the algebraic core of the SVD — the next lesson's main tool.

Knowledge check#

Browser lab#

Check positive definiteness via eigenvalues and plot the quadratic form's level set.

import numpy as np
import matplotlib.pyplot as plt

A = np.array([[2., -1.],
              [-1., 2.]])

eigvals = np.linalg.eigvalsh(A)
print("eigenvalues =", eigvals)
print("positive definite:", np.all(eigvals > 0))

theta = np.linspace(0, 2 * np.pi, 200)
circle = np.array([np.cos(theta), np.sin(theta)])

vals, vecs = np.linalg.eigh(A)
ellipse = vecs @ np.diag(1 / np.sqrt(vals)) @ circle

fig, ax = plt.subplots(figsize=(5, 5))
ax.plot(ellipse[0], ellipse[1])
ax.set_aspect("equal")
ax.set_title(r"Level set $x^TAx = 1$")
plt.show()

np.linalg.eigvalsh / eigh are the symmetric (Hermitian) variants of the eigensolvers — they return real eigenvalues sorted ascending and an orthogonal eigenvector matrix. The construction $Q\,\operatorname{diag}(1/\sqrt{\lambda_i})\,(\cos\theta, \sin\theta)$ maps the unit circle to the ellipse $x^TAx = 1$: half-axis lengths are $1/\sqrt{\lambda_i}$ along the eigenvectors, matching the geometric picture above.

Quiz#