Deep Learning and Effective Correlation Spaces

deep-learning-theoryweightwatcherpower-lawsspectral-analysisneural-networks

Abstraction: Power-law ESD tails in DNN layers indicate optimal generalization via SETOL

Key points:

The WeightWatcher open-source tool (86K+ downloads, featured in Nature Communications) diagnoses DNN model quality layer-by-layer without needing training or test data
SETOL (Semi-Empirical Theory of Learning) expresses model quality as an approximate free energy via an HCIZ integral; the weightwatcher alpha-hat metric approximates the log model quality when the model is well-trained
Key theoretical claim: for well-trained layers, weight matrix correlations concentrate into a lower-rank Effective Correlation Space characterized by a Volume-Preserving Transformation (Jacobian determinant = 1)
Two independent methods identify the power-law tail of the Empirical Spectral Density: MLE Clauset fit (red line) and the det(X)=1 constraint (purple line); when they coincide, the theory holds and the layer PL exponent alpha is approximately 2.0
HTSR theory (published in JMLR) states the ESD power-law tail contains the dominant eigencomponents enabling generalization; alpha ≈ 2 corresponds to optimal learning
Empirically verified on ALBERT (all four sizes) and VGG19; tool invocable with three lines: import weightwatcher, watcher.analyze(model, detX=True)

Connections: Weightwatcher · Deep Learning Theory · Neural Network Analysis · Power Laws