Principal Component Analysis (PCA)

Dimensionality Reduction through Variance Maximization

A fundamental technique for reducing data dimensionality while preserving maximum variance

Key Concept: Find orthogonal directions of maximum variance in high-dimensional data
Learning Goals:
  • Understand the curse of dimensionality
  • Master the mathematical foundations of PCA
  • Interpret principal components geometrically
  • Apply PCA for visualization and compression

The Curse of Dimensionality

Why High Dimensions Are Problematic

  • Volume Explosion: Unit hypercube volume remains 1, but most points lie near edges
  • Distance Concentration: All pairwise distances become similar
  • Sparsity: Data points become isolated in high-dimensional space
  • Computational Cost: $O(d^3)$ algorithms become intractable
Volume of d-dimensional unit sphere:
$V_d = \frac{\pi^{d/2}}{\Gamma(d/2 + 1)}$
Real Examples: Images (784D), Gene expression (20,000D), Text (vocabulary size)

Mathematical Foundation of PCA

Covariance Matrix

$C = \frac{1}{n-1}X^T X$
(for centered data $X$)

Captures linear relationships between features

Eigendecomposition

$C = Q\Lambda Q^T$
$C q_i = \lambda_i q_i$
  • $\lambda_i$: variance along PC $i$
  • $q_i$: direction of PC $i$
  • $||q_i|| = 1$, $q_i^T q_j = 0$ for $i \neq j$

Geometric Interpretation

Principal Components

  • PC1: Direction of maximum variance
  • PC2: Orthogonal direction, next highest variance
  • PCk: Orthogonal to all previous, remaining variance

Dimensionality Reduction

$Y = X Q_k$
Project onto first $k$ components

Information loss in discarded components

PCA Algorithm Steps

Data Preprocessing

Step 1: Center the data
$X_{centered} = X - \mu$

Step 2: Optional scaling
$X_{scaled} = \frac{X_{centered}}{\sigma}$

Eigendecomposition

Step 3: Compute covariance
$C = \frac{1}{n-1}X^T X$

Step 4: Find eigenvalues/vectors
$C = Q\Lambda Q^T$

Step 5: Sort by eigenvalue
$\lambda_1 \geq \lambda_2 \geq ... \geq \lambda_d$

Choosing Number of Components

Explained Variance Ratio

$EVR_i = \frac{\lambda_i}{\sum_{j=1}^d \lambda_j}$

Proportion of variance explained by component $i$

Cumulative Variance

$CVR_k = \sum_{i=1}^k EVR_i$

Selection Criteria

  • Threshold: 80-95% cumulative variance
  • Elbow Method: Look for "knee" in scree plot
  • Kaiser Rule: Keep $\lambda_i > 1$ (standardized data)
  • Cross-validation: Optimize for downstream task

Implementation Considerations

Computational Approaches

Full Eigendecomposition
• Complexity: $O(d^3)$
• Exact solution
• Memory: $O(d^2)$

SVD Approach
• $X = U\Sigma V^T$
• PCs are columns of $V$
• More numerically stable

Randomized PCA
• Approximate top-k components
• $O(ndk)$ complexity
• Scalable to large datasets
Key Decision: Standardization affects results significantly - use when features have different units/scales

Reconstruction and Information Loss

Data Reconstruction

$\tilde{X} = X Q_k Q_k^T$

Approximate original data using first $k$ components

Reconstruction Error

$Error = ||X - \tilde{X}||_F^2 = \sum_{i=k+1}^d \lambda_i$

Information Preservation

  • Perfect reconstruction when $k = d$
  • Error equals sum of discarded eigenvalues
  • PCA minimizes reconstruction error (optimal linear projection)
  • Trade-off: compression vs. information loss

Practical Example: Iris Dataset

Original Features (4D)

  • Sepal Length
  • Sepal Width
  • Petal Length
  • Petal Width

PC Interpretation

PC1: Overall size
PC2: Shape contrast
Together explain ~95% variance

Results

  • Dimensionality: 4D → 2D
  • Variance Retained: 95.8%
  • Class Separation: Well preserved
  • Visualization: Clear clustering
PC1: 72.9% variance
PC2: 22.9% variance

PCA: Advantages and Limitations

✅ Advantages

  • Optimal: Minimizes reconstruction error
  • Interpretable: Components show variance directions
  • Uncorrelated: Orthogonal components
  • Efficient: Fast computation with SVD
  • Versatile: Preprocessing, visualization, compression
  • Noise Reduction: Filters out low-variance directions
Best Use Cases:
• High-dimensional visualization
• Noise reduction
• Data compression
• Computational speedup

❌ Limitations

  • Linear Only: Cannot capture nonlinear relationships
  • Variance ≠ Information: May discard useful low-variance features
  • Scaling Sensitive: Results depend on feature scales
  • Interpretation: Components may lack intuitive meaning
  • Outlier Sensitive: Extreme values affect directions
  • Global Method: Same transformation for all data points
Alternatives:
• t-SNE, UMAP (nonlinear)
• ICA (non-Gaussian)
• Autoencoders (neural)

Key Takeaways

🎯 Core Concepts

  • PCA finds orthogonal directions of maximum variance
  • Eigenvalues quantify component importance
  • Trade-off between dimensionality and information retention
  • Preprocessing (centering/scaling) critically affects results

🔧 Practical Guidelines

  • Always center data ($\mu = 0$)
  • Standardize when features have different scales
  • Use scree plot + variance threshold for component selection
  • Consider SVD for numerical stability
  • Validate with downstream task performance

🎨 Applications

  • Visualization: High-D → 2D/3D plots
  • Compression: Reduce storage/computation
  • Preprocessing: Noise reduction, decorrelation
  • Anomaly Detection: Reconstruction error
  • Feature Engineering: Principal component features
Remember: PCA is a linear method - consider nonlinear alternatives for complex manifold structures
1 / 11