Making sense of principal component analysis, eigenvectors & eigenvalues

Abstraction: Geometric and computational walkthrough of PCA via eigendecomposition and SVD

Key points:

• PCA computes the covariance matrix of scaled data, then finds eigenvectors (principal components) and eigenvalues; eigenvalues indicate variance explained (e.g., PC1 64.8%, PC2 35.2% in the worked example)
• The score matrix is computed as scaled_data × eigenvector_matrix; variance of each PC column equals its eigenvalue exactly
• PCA is equivalent to SVD on the scaled data matrix: prcomp() in R uses SVD internally; singular values relate to eigenvalues by (d/√(n-1))²
• Original data is recovered by multiplying the score matrix by the inverse of the eigenvector matrix (rotation back)
• PCA differs from OLS: PCA minimizes orthogonal distances to the component (L2-norm of projections), while OLS minimizes vertical residuals — they produce different hyperplanes
• Loadings (eigenvectors scaled by eigenvalues) indicate the contribution and direction of original variables to each component; visualized via biplots

Connections: Principal Component Analysis · Dimensionality Reduction · Eigenvectors · Linear Algebra

Source: http://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues