Singular Value Decomposition Part 1: Perspectives on Linear Algebra

svdlinear-algebramatrix-factorizationdimensionality-reductiondata-science

Abstraction: Intuitive motivation for SVD as data approximation via matrix factorization

Key points:

SVD factorizes an m×n matrix A as UΣV^T where U and V are orthogonal and Σ is diagonal; the key insight is this finds the best rank-k approximation of A (Eckart-Young theorem)
Two conflicting matrix interpretations — as a linear map vs. as organized data — collide in SVD; data columns are the images of basis vectors under the linear map
SVD is equivalent to iteratively solving a greedy optimization: "what is the best-fit line, then the next best ignoring the first, etc." — surprisingly the greedy approach is globally optimal
Low-rank approximation "smooths noise": real data is almost always full-rank due to noise, but SVD finds the closest low-rank surrogate by a spectral-norm measure
Geometric interpretation: any linear map decomposes into rotation, coordinate rescaling, then rotation; the approximation layers (singular vectors/values) are ordered strongest to weakest
Applications: recommendation systems, PageRank, regression, PCA; movie-rating example illustrates "idealized critics" and "idealized movies" as singular vectors

Connections: Singular Value Decomposition · Linear Algebra · Dimensionality Reduction · Matrix Factorization