Simpson's Paradox and Deep Learning Metrics with Weightwatcher

Abstraction: WeightWatcher power-law metrics resolving Simpson's paradox in DNN evaluation

Key points:

• WeightWatcher is an open-source Python tool that predicts DNN test accuracy trends without accessing test data, using only layer weight matrices
• Two key metrics: alpha (power-law exponent per layer) for comparing models with different hyperparameters; alpha_weighted (weighted average of log spectral norm) for comparing architectures of different depths (e.g., VGG11 to VGG19)
• Simpson's paradox discovered: average log spectral norm is correlated with test error across depths but anti-correlated within a model group varying hyperparameters — making it unsuitable as a general metric alone
• alpha_weighted corrects this by weighting the spectral norm by the per-layer alpha, resolving the paradox and enabling cross-model comparison
• Spectral Norm Regularization bounds DNN robustness to input perturbations; WeightWatcher shifts focus from worst-case bounds to average-case generalization trends

Connections: Weightwatcher · Spectral Norms · Neural Network Analysis · Simpsons Paradox

Source: https://calculatedcontent.com/2020/11/26/simpsons-paradox-and-deep-learning-metrics-with-weightwatcher/