Skip to main content

This Week's Best Picks from Amazon

Please see more curated items that we picked from Amazon here .

Orthogonality and Orthogonal Projections

  • Define orthogonal vectors and orthonormal sets using the inner product.
  • Understand the orthogonal complement of a subspace.
  • Derive the formula for projecting a vector onto a line.
  • Generalize projection to an arbitrary subspace using the projection matrix.
  • See how orthogonal projection is exactly what the least squares method computes.

Orthogonal Vectors and Sets

Two vectors \( \mathbf{u} \) and \( \mathbf{v} \) are orthogonal when their inner product is zero:

\[ \langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^T \mathbf{v} = 0 \]

A set of vectors is orthogonal if every pair in it is orthogonal, and orthonormal if it is orthogonal and every vector also has unit length, \( \|\mathbf{v}_i\| = 1 \). Orthonormal bases are the easiest to work with: coordinates in that basis are found by simple inner products instead of solving a linear system.

Orthogonal Complement

The orthogonal complement of a subspace \( W \subseteq \mathbb{R}^n \), written \( W^{\perp} \), is the set of all vectors orthogonal to every vector in \( W \):

\[ W^{\perp} = \{ \mathbf{v} \in \mathbb{R}^n \mid \mathbf{v}^T \mathbf{w} = 0 \text{ for all } \mathbf{w} \in W \} \]

This is exactly the relationship we flagged in Rank, Null Space, and the Four Fundamental Subspaces: for any matrix \( A \), the null space and row space are orthogonal complements of each other, and the column space and left null space are orthogonal complements:

\[ N(A) = C(A^T)^{\perp}, \qquad N(A^T) = C(A)^{\perp} \]

Projection onto a Line

The orthogonal projection of a vector \( \mathbf{b} \) onto the line spanned by \( \mathbf{a} \) is the point on that line closest to \( \mathbf{b} \). It's given by:

\[ \text{proj}_{\mathbf{a}}(\mathbf{b}) = \frac{\mathbf{a}^T \mathbf{b}}{\mathbf{a}^T \mathbf{a}} \, \mathbf{a} \]

The residual \( \mathbf{b} - \text{proj}_{\mathbf{a}}(\mathbf{b}) \) is orthogonal to \( \mathbf{a} \) — that orthogonality is precisely what makes the projection the closest point on the line.

Projection onto a Subspace

To project onto a higher-dimensional subspace \( W = C(A) \) spanned by the columns of a matrix \( A \), the projection is:

\[ \hat{\mathbf{b}} = A(A^T A)^{-1} A^T \mathbf{b} \]

The matrix \( P = A(A^T A)^{-1} A^T \) is called the projection matrix onto \( C(A) \). It satisfies two defining properties: \( P^2 = P \) (projecting twice does nothing new) and \( P^T = P \) (it's symmetric).

Connection to Least Squares

This is exactly the machinery behind the Least Squares Method covered earlier in this series. When \( A\mathbf{x} = \mathbf{b} \) has no exact solution because \( \mathbf{b} \) isn't in \( C(A) \), the best approximate solution replaces \( \mathbf{b} \) with its orthogonal projection \( \hat{\mathbf{b}} \) onto \( C(A) \) — the closest point in the column space. Solving \( A\mathbf{x} = \hat{\mathbf{b}} \) gives exactly the normal equations \( A^T A \mathbf{x} = A^T \mathbf{b} \) from that lesson. Least squares, in other words, is orthogonal projection.

Examples

Example 1: Project \( \mathbf{b} = \begin{bmatrix} 3 \\ 4 \end{bmatrix} \) onto the line spanned by \( \mathbf{a} = \begin{bmatrix} 1 \\ 0 \end{bmatrix} \).

\[ \text{proj}_{\mathbf{a}}(\mathbf{b}) = \frac{\mathbf{a}^T \mathbf{b}}{\mathbf{a}^T \mathbf{a}} \, \mathbf{a} = \frac{3}{1} \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 3 \\ 0 \end{bmatrix} \]

Example 2: Verify the residual is orthogonal to \( \mathbf{a} \): \( \mathbf{b} - \text{proj}_{\mathbf{a}}(\mathbf{b}) = \begin{bmatrix} 0 \\ 4 \end{bmatrix} \), and indeed \( \begin{bmatrix} 1 \\ 0 \end{bmatrix}^T \begin{bmatrix} 0 \\ 4 \end{bmatrix} = 0 \). ✓

Exercises

  • Question 1: Are \( \mathbf{u} = \begin{bmatrix} 2 \\ 1 \end{bmatrix} \) and \( \mathbf{v} = \begin{bmatrix} -1 \\ 2 \end{bmatrix} \) orthogonal?
  • Question 2: Project \( \mathbf{b} = \begin{bmatrix} 5 \\ 1 \end{bmatrix} \) onto the line spanned by \( \mathbf{a} = \begin{bmatrix} 1 \\ 1 \end{bmatrix} \).
  • Question 3: What two properties define a projection matrix \( P \)?

  • Answer 1: \( \mathbf{u}^T \mathbf{v} = (2)(-1) + (1)(2) = 0 \), so yes, they're orthogonal.
  • Answer 2: \( \text{proj}_{\mathbf{a}}(\mathbf{b}) = \frac{6}{2}\begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 3 \\ 3 \end{bmatrix} \).
  • Answer 3: \( P^2 = P \) (idempotent) and \( P^T = P \) (symmetric).

Popular posts from this blog

Eigenvalues and Eigenvectors

Lesson Objectives ▼ Understand the definition of eigenvalues and eigenvectors. Learn how to compute eigenvalues and eigenvectors. Explore the characteristic equation. Interpret eigenvalues and eigenvectors in transformations. Understand diagonalization and its applications. Lesson Outline ▼ Definition of Eigenvalues and Eigenvectors Characteristic Equation Diagonalization Examples Definition of Eigenvalues and Eigenvectors In a linear transformation, an eigenvector is a nonzero vector that only changes by a scalar factor when the transformation is applied. For a square matrix \( A \), an eigenvector \( \mathbf{v} \) and its corresponding eigenvalue \( \lambda \) satisfy: \[ A\mathbf{v} = \lambda \mathbf{v} \] \( A \) is an \( n \times n \) matrix. \( \mathbf{v} \neq 0 \) is an eigenvector. \( \lambda \) is a scalar eigenvalue. Characteristic Equation ...
Read more

Inner Product Spaces

Lesson Objectives ▼ Understand the definition of inner product spaces. Compute norms, angles, and distances in inner product spaces. Explore the concept of orthogonality. Learn the Gram-Schmidt orthogonalization process. Use inner products for projections and least squares approximation. Lesson Outline ▼ Definition of Inner Product Spaces Norms, Angles, and Orthogonality Gram-Schmidt Orthogonalization Projections and Least Squares Examples Definition of Inner Product Spaces An inner product space is a vector space with an operation called the **inner product**, which measures similarity between vectors. For a real vector space, the inner product of two vectors \( \mathbf{u}, \mathbf{v} \) is: \[ \langle \mathbf{u}, \mathbf{v} \rangle = u_1 v_1 + u_2 v_2 + \dots + u_n v_n \] For a complex vector space, the inner product is: \[ \langle \mathbf{u}, \mathbf{v} \ran...
Read more

Sets and Binary Operations

Lesson Objectives ▼ Understand the concept of a set and its basic operations. Learn about binary operations and their properties. Explore examples of binary operations. Lesson Outline ▼ Introduction to Sets Basic Set Operations Binary Operations Introduction to Sets A set is a collection of distinct objects, considered as an object in its own right. For example, the set of natural numbers \( \{1, 2, 3, \ldots\} \). Sets can be represented using curly braces, e.g., \( \{a, b, c\} \). Basic Set Operations Union: The union of sets \(A\) and \(B\) is the set of elements that are in \(A\), \(B\), or both. Denoted as \(A \cup B\). Intersection: The intersection of sets \(A\) and \(B\) is the set of elements that are in both \(A\) and \(B\). Denoted as \(A \cap B\). Difference: The difference of sets \(A\) and \(B\) is the set of elements that are in \(A\) but not in \(B\). Denoted as \(A...
Read more