- Define orthogonal vectors and orthonormal sets using the inner product.
- Understand the orthogonal complement of a subspace.
- Derive the formula for projecting a vector onto a line.
- Generalize projection to an arbitrary subspace using the projection matrix.
- See how orthogonal projection is exactly what the least squares method computes.
Orthogonal Vectors and Sets
Orthogonality
Two vectors \( \mathbf{u} \) and \( \mathbf{v} \) are orthogonal when their inner product is zero:
\[ \langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^T \mathbf{v} = 0. \]Geometrically, this means the two vectors meet at a right angle. A set of vectors is orthogonal when this holds for every pair of distinct vectors in it, that is, \( \mathbf{v}_i^T \mathbf{v}_j = 0 \) whenever \( i \neq j \).
The vectors
\[ \mathbf{v}_1 = \begin{pmatrix} 2 \\ 0 \end{pmatrix}, \quad \mathbf{v}_2 = \begin{pmatrix} 0 \\ 3 \end{pmatrix} \]form an orthogonal set, since \( \mathbf{v}_1^T \mathbf{v}_2 = (2)(0) + (0)(3) = 0 \). One lies along the horizontal axis and the other along the vertical axis, so they are perpendicular. By contrast, the vectors
\[ \mathbf{u}_1 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}, \quad \mathbf{u}_2 = \begin{pmatrix} 1 \\ 0 \end{pmatrix} \]are not orthogonal, because \( \mathbf{u}_1^T \mathbf{u}_2 = (1)(1) + (1)(0) = 1 \neq 0 \). Geometrically, \( \mathbf{u}_1 \) points diagonally at \( 45^\circ \) to \( \mathbf{u}_2 \), so the angle between them is not a right angle.
Orthonormality
Orthogonality says nothing about length. When we also require every vector to have length one, we arrive at a stronger and more useful notion. A set of vectors is orthonormal if the vectors are orthogonal and each has unit length, \( \|\mathbf{v}_i\| = 1 \). Both conditions combine into a single statement:
\[ \mathbf{v}_i^T \mathbf{v}_j = \begin{cases} 1 & \text{if } i = j, \\ 0 & \text{if } i \neq j. \end{cases} \]Orthonormal bases are the easiest to work with: coordinates in that basis are found by simple inner products instead of solving a linear system.
The standard basis of the plane is the prototype:
\[ \mathbf{e}_1 = \begin{pmatrix} 1 \\ 0 \end{pmatrix}, \quad \mathbf{e}_2 = \begin{pmatrix} 0 \\ 1 \end{pmatrix}. \]Their inner product is \( 0 \) and each has length \( 1 \), so the set is orthonormal.
Now return to the orthogonal pair \( \mathbf{v}_1 = (2,0)^T \) and \( \mathbf{v}_2 = (0,3)^T \) from before. They are still perpendicular, but their lengths are \( \|\mathbf{v}_1\| = 2 \) and \( \|\mathbf{v}_2\| = 3 \), so the set is orthogonal without being orthonormal.
Orthogonal Complement
The orthogonal complement of a subspace \( W \subseteq \mathbb{R}^n \), written \( W^{\perp} \), is the set of all vectors orthogonal to every vector in \( W \):
\[ W^{\perp} = \{ \mathbf{v} \in \mathbb{R}^n \mid \mathbf{v}^T \mathbf{w} = 0 \text{ for all } \mathbf{w} \in W \} \]This is exactly the relationship we flagged in Rank, Null Space, and the Four Fundamental Subspaces: for any matrix \( A \), the null space and row space are orthogonal complements of each other, and the column space and left null space are orthogonal complements:
\[ N(A) = C(A^T)^{\perp}, \qquad N(A^T) = C(A)^{\perp} \]Projection onto a Line
The orthogonal projection of a vector \( \mathbf{b} \) onto the line spanned by \( \mathbf{a} \) is the point on that line closest to \( \mathbf{b} \). It's given by:
\[ \text{proj}_{\mathbf{a}}(\mathbf{b}) = \frac{\mathbf{a}^T \mathbf{b}}{\mathbf{a}^T \mathbf{a}} \, \mathbf{a} \]The residual \( \mathbf{b} - \text{proj}_{\mathbf{a}}(\mathbf{b}) \) is orthogonal to \( \mathbf{a} \) — that orthogonality is precisely what makes the projection the closest point on the line.
Projection onto a Subspace
To project onto a higher-dimensional subspace \( W = C(A) \) spanned by the columns of a matrix \( A \), the projection is:
\[ \hat{\mathbf{b}} = A(A^T A)^{-1} A^T \mathbf{b} \]The matrix \( P = A(A^T A)^{-1} A^T \) is called the projection matrix onto \( C(A) \). It satisfies two defining properties: \( P^2 = P \) (projecting twice does nothing new) and \( P^T = P \) (it's symmetric).
Connection to Least Squares
This is exactly the machinery behind the Least Squares Method covered earlier in this series. When \( A\mathbf{x} = \mathbf{b} \) has no exact solution because \( \mathbf{b} \) isn't in \( C(A) \), the best approximate solution replaces \( \mathbf{b} \) with its orthogonal projection \( \hat{\mathbf{b}} \) onto \( C(A) \) — the closest point in the column space. Solving \( A\mathbf{x} = \hat{\mathbf{b}} \) gives exactly the normal equations \( A^T A \mathbf{x} = A^T \mathbf{b} \) from that lesson. Least squares, in other words, is orthogonal projection.
Examples
Example 1: Project \( \mathbf{b} = \begin{bmatrix} 3 \\ 4 \end{bmatrix} \) onto the line spanned by \( \mathbf{a} = \begin{bmatrix} 1 \\ 0 \end{bmatrix} \).
\[ \text{proj}_{\mathbf{a}}(\mathbf{b}) = \frac{\mathbf{a}^T \mathbf{b}}{\mathbf{a}^T \mathbf{a}} \, \mathbf{a} = \frac{3}{1} \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 3 \\ 0 \end{bmatrix} \]Example 2: Verify the residual is orthogonal to \( \mathbf{a} \): \( \mathbf{b} - \text{proj}_{\mathbf{a}}(\mathbf{b}) = \begin{bmatrix} 0 \\ 4 \end{bmatrix} \), and indeed \( \begin{bmatrix} 1 \\ 0 \end{bmatrix}^T \begin{bmatrix} 0 \\ 4 \end{bmatrix} = 0 \). ✓
Exercises
- Question 1: Are \( \mathbf{u} = \begin{bmatrix} 2 \\ 1 \end{bmatrix} \) and \( \mathbf{v} = \begin{bmatrix} -1 \\ 2 \end{bmatrix} \) orthogonal?
- Question 2: Project \( \mathbf{b} = \begin{bmatrix} 5 \\ 1 \end{bmatrix} \) onto the line spanned by \( \mathbf{a} = \begin{bmatrix} 1 \\ 1 \end{bmatrix} \).
- Question 3: What two properties define a projection matrix \( P \)?
- Answer 1: \( \mathbf{u}^T \mathbf{v} = (2)(-1) + (1)(2) = 0 \), so yes, they're orthogonal.
- Answer 2: \( \text{proj}_{\mathbf{a}}(\mathbf{b}) = \frac{6}{2}\begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 3 \\ 3 \end{bmatrix} \).
- Answer 3: \( P^2 = P \) (idempotent) and \( P^T = P \) (symmetric).