Allen Tran
probabilistic-pcadimensionality-reductionmissing-datamacroeconomics
Abstraction: Probabilistic PCA with EM handles missing data in macroeconomic series
Key points:
- Standard PCA fails with missing data; Probabilistic PCA (PPCA) uses an iterative EM algorithm to alternately interpolate missing values and update components
- Applied to 14,000+ monthly CPS macroeconomic series where 65% of data from series dating back to 1948 are missing (as of 2007)
- 93 components (< 0.6% of dataset size) explain over 90% of variance in the full dataset
- PPCA performs multivariate interpolation using cross-series information, unlike univariate methods (mean imputation, etc.)
- Overfitting risk: fitting too many components causes overfitting in the interpolated data
- Author's custom implementation available at github.com/allentran/pca-magic since scikit-learn's PPCA doesn't handle missing data
Connections: Allen Tran · Probabilistic Pca · Dimensionality Reduction · Time Series Analysis
Source: http://allentran.github.io/ppca