Week 12: Symmetric and Positive Definite Matrices

The Spectral Theorem#

A real matrix $A$ is symmetric when $A = A^T$ — every entry equals its transpose partner, so $a_{ij} = a_{ji}$. Symmetric matrices are the ones that show up in applications: covariance matrices, Hessians of smooth functions, graph Laplacians, and Gram matrices of the form $B^TB$ all land here.

The spectral theorem is the central fact:

Every real symmetric $n \times n$ matrix $A$ has $n$ real eigenvalues $\lambda_1,\ldots,\lambda_n$ (counting multiplicity) and an orthonormal basis of eigenvectors $q_1,\ldots,q_n$. In matrix form,

where $\Lambda = \operatorname{diag}(\lambda_1,\ldots,\lambda_n)$ and $Q$ is an orthogonal matrix ($Q^TQ = I$, so $Q^{-1} = Q^T$) whose columns are the eigenvectors.

Compare with ordinary diagonalization $A = S\Lambda S^{-1}$ from Week 10. Symmetry upgrades two things at once:

1. All eigenvalues are real — no complex-conjugate pairs.
2. The eigenvector matrix can be chosen orthogonal — $S^{-1}$ is simply $S^T$, and the change of coordinates is a pure rotation/reflection (an isometry of $\mathbb{R}^n$).

Why eigenvectors for distinct eigenvalues are orthogonal. Suppose $Aq_1 = \lambda_1 q_1$, $Aq_2 = \lambda_2 q_2$, with $\lambda_1 \neq \lambda_2$ and $A = A^T$. Take the inner product:

So $(\lambda_1 - \lambda_2)\, q_2^T q_1 = 0$. Because $\lambda_1 \neq \lambda_2$, we must have $q_2^T q_1 = 0$: the eigenvectors are orthogonal. When eigenvalues repeat, you can still orthonormalize within each eigenspace (Gram–Schmidt from Week 7) and keep different eigenspaces orthogonal. The result is always an orthonormal basis of $\mathbb{R}^n$.

Worked check. Let

Trace $4$, determinant $3$, so eigenvalues solve $\lambda^2 - 4\lambda + 3 = 0$: $\lambda_1 = 3$, $\lambda_2 = 1$. For $\lambda = 3$:

For $\lambda = 1$:

Indeed $q_1^T q_2 = 0$, and

(You can verify $Q^TQ = I$ and $Q\Lambda Q^T = A$ by direct multiplication.)

Quadratic Forms#

A quadratic form is a scalar function of a vector built from a symmetric matrix:

In components for $n = 2$ with $A = \begin{pmatrix} a & b \\ b & c \end{pmatrix}$ this is

(The off-diagonal $b$ contributes twice because $a_{12}$ and $a_{21}$ both equal $b$.) Quadratic forms measure energy, squared length after a linear map ($\|Bx\|^2 = x^T(B^TB)x$), and second-order Taylor terms of a smooth function near a critical point.

Diagonalization simplifies the form. Write $A = Q\Lambda Q^T$ and change variables to the eigen-coordinates $y = Q^T x$ (so $x = Qy$). Then

The cross terms vanish, leaving a pure weighted sum of squares. That is why the spectral theorem is the engine of everything that follows — positive definiteness, ellipses, and (next week) the SVD all reduce to the signs and sizes of the $\lambda_i$.

Example. For the matrix above with $\lambda = 3,1$,

in the $Q$-coordinates. Along the first eigen-direction the form grows thrice as fast as along the second.

Positive Definite Matrices#

A symmetric matrix $A$ is positive definite (often written $A \succ 0$) when

If the inequality is $\ge 0$ instead of $> 0$, $A$ is positive semidefinite. Negative definite means $x^TAx < 0$ for all $x \neq 0$ (equivalently, $-A$ is positive definite).

Positive definite matrices are the multivariable analogue of a positive number: they define genuine energies, strict convexity of $x^TAx/2$, and unique minima of associated least-squares problems (see Optimization Week 2: Convex Sets and Functions for the convexity view). Three classical tests decide positive definiteness — all equivalent for real symmetric $A$ (Strang, §6.4).

Test 1 — Eigenvalues#

From the spectral form $x^TAx = \sum \lambda_i y_i^2$ with $y = Q^Tx$ and $\|y\| = \|x\|$ (because $Q$ is orthogonal), we get:

• $A$ is positive definite $\Longleftrightarrow$ every eigenvalue $\lambda_i > 0$.
• $A$ is positive semidefinite $\Longleftrightarrow$ every $\lambda_i \ge 0$.
• $A$ is indefinite if some $\lambda_i > 0$ and some $\lambda_j < 0$ (the form takes both positive and negative values).

This is usually the conceptual test: stretch factors in every eigen-direction must be positive.

Test 2 — Pivots#

Eliminate $A$ by Gaussian elimination without row exchanges. The pivots $d_1, d_2, \ldots, d_n$ on the diagonal of the upper-triangular factor are all positive if and only if $A$ is positive definite; equivalently, the LDLLDL^T (symmetric matrix factorization) factorization $A = LDL^T$ has a positive diagonal $D$.

Test 3 — Leading principal minors (determinants)#

Let $A_k$ be the $k \times k$ submatrix in the upper-left corner of $A$ (its first $k$ rows and columns). The leading principal minors are $\det(A_1),\ldots,\det(A_n)$. Sylvester's criterion:

For a $2 \times 2$ matrix $\begin{pmatrix} a & b \\ b & c \end{pmatrix}$ this is simply $a > 0$ and $ac - b^2 > 0$.

Worked example. Take again

• Eigenvalues: $\lambda = 3$ and $\lambda = 1$, both positive $\Rightarrow$ positive definite.
• Pivots: first pivot $2 > 0$; after eliminating, the Schur complement is $2 - (-1)^2/2 = 3/2 > 0$. Both pivots positive.
• Leading determinants: $\det(A_1) = 2 > 0$ and $\det(A) = 4 - 1 = 3 > 0$.

All three tests agree. Contrast with $B = \begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix}$: eigenvalues $3$ and $-1$, so $B$ is indefinite — $\det(B) = -3 < 0$ already fails the leading-determinant test.

Why three tests? Eigenvalues are conceptual. Pivots are computational during elimination. Leading determinants are a quick hand check for small $n$. Use whichever is handy; they always give the same yes/no answer for symmetric $A$.

Geometric Picture#

Fix a positive definite $A$ and a positive constant $c$. The level set

is a closed bounded surface around the origin. In the eigen-coordinates $y = Q^T x$ the equation becomes

or equivalently

That is the standard equation of an ellipse when $n = 2$, and an ellipsoid when $n \ge 3$. The geometry is completely determined by the spectral data:

Large $\lambda_i$ means a short axis: the quadratic form grows quickly in that direction, so the level set $x^TAx = c$ is reached sooner. Small $\lambda_i$ means a long axis. When one eigenvalue is much larger than another, the ellipse is elongated along the small-eigenvalue direction.

The sign pattern of the eigenvalues decides the shape. Both positive gives a closed ellipse (the energy grows in every direction); opposite signs give hyperbolas, along which the form is positive in some directions and negative in others.

If $A$ is not positive definite. Zero eigenvalues open the level set into a cylinder, unbounded along the null direction. Opposite-sign eigenvalues produce hyperbolas: the form is indefinite, so the “energy” $x^TAx$ is positive along some directions and negative along others.

Preview of Week 13. For any matrix $B$ (not necessarily square), the Gram matrix $A = B^TB$ is always symmetric and positive semidefinite. Its eigenvalues are the squared singular values of $B$, and its eigenvectors are the right singular vectors. The spectral theorem for $A$ is the algebraic core of the SVD — the next lesson's main tool.

Knowledge check#

Browser lab: positive definiteness#

Check positive definiteness via eigenvalues and plot the quadratic form's level set.

import numpy as np
import matplotlib.pyplot as plt

A = np.array([[2., -1.],
              [-1., 2.]])

eigvals = np.linalg.eigvalsh(A)
print("eigenvalues =", eigvals)
print("positive definite:", np.all(eigvals > 0))

theta = np.linspace(0, 2 * np.pi, 200)
circle = np.array([np.cos(theta), np.sin(theta)])

vals, vecs = np.linalg.eigh(A)
ellipse = vecs @ np.diag(1 / np.sqrt(vals)) @ circle

fig, ax = plt.subplots(figsize=(5, 5))
ax.plot(ellipse[0], ellipse[1])
ax.set_aspect("equal")
ax.set_title(r"Level set $x^TAx = 1$")
plt.show()

print("Notice: both eigenvalues are positive, so 'positive definite: True', and the level set is a tilted ellipse.")

np.linalg.eigvalsh / eigh are the symmetric (Hermitian) variants of the eigensolvers — they return real eigenvalues sorted ascending and an orthogonal eigenvector matrix. The construction $Q\,\operatorname{diag}(1/\sqrt{\lambda_i})\,(\cos\theta, \sin\theta)$ maps the unit circle to the ellipse $x^TAx = 1$: half-axis lengths are $1/\sqrt{\lambda_i}$ along the eigenvectors, matching the geometric picture above.

Done when#

You can state the spectral theorem, test a symmetric matrix for positive definiteness by eigenvalues, pivots, or leading determinants, and read a quadratic form's level set as an ellipse or ellipsoid. Check it against the browser lab: for $A=\begin{pmatrix}2&-1\\-1&2\end{pmatrix}$ the printed eigenvalues are 1. and 3. and positive definite: True, and you can give the ellipse half-axis lengths $1/\sqrt{3}$ and $1$.

Further Reading#

• Strang, G. Introduction to Linear Algebra, 6th ed. — §6.2 Symmetric Matrices, §6.4 Positive Definite Matrices
• Trefethen, L.N. & Bau, D. Numerical Linear Algebra — Lecture 25 (Spectral Theorem)