Week 4: Matrix Structures in Machine Learning

Overview

This week explores four fundamental matrix structures, their historical emergence, and their applications in modern machine learning. We focus on understanding when and why each structure is useful rather than mathematical details.

Learning Objectives

• Identify and understand different matrix structures
• Match data types to appropriate matrix representations
• Recognize which algorithms work best with each structure
• Understand the historical evolution of matrix applications

1. Tall and Skinny Matrices (1960s-1970s)

• Structure
  • More rows than columns (m >> n)
  • Typical dimensions: 10000 × 100
  • Dense or sparse formats
• Applications
  • Survey data analysis
  • Medical trials (patients × features)
  • Sensor readings over time
• Suitable Algorithms
  • Principal Component Analysis
  • Linear Regression
  • Factor Analysis

2. Short and Fat Matrices (1980s-1990s)

• Structure
  • More columns than rows (m << n)
  • Typical dimensions: 100 × 10000
  • Often sparse
• Applications
  • Text document analysis
  • Recommendation systems
• Suitable Algorithms
  • Random Forest
  • LASSO regression
  • Ridge regression
  • Compressed sensing

3. Square Matrices (Classical-Present)

• Structure
  • Equal rows and columns (n × n)
  • Dense representation
• Applications
  • Social network connections
  • Distance/similarity matrices
  • State transition matrices
• Suitable Algorithms
  • Graph algorithms
  • Markov chains
  • Eigenvalue decomposition

4. Very Tall and Skinny Matrices (2010s-Present)

• Structure
  • Extremely more rows than columns
  • Typical dimensions: 1M × 100
  • Batch processing structure
• Applications
  • Deep learning datasets
  • Large-scale image processing
  • Time-series prediction
• Suitable Algorithms
  • Stochastic gradient descent
  • Mini-batch processing
  • Neural network training