A New Study by Google and DeepMind Introduces Geometric Complexity (GC) for Neural Network Analysis and Understanding of Deep Learning Models

Abstraction: Geometric Complexity measure explains regularization and double-descent in neural networks

Key points:

• Geometric Complexity (GC) is a new model complexity measure purpose-built for deep neural networks, addressing limitations of classical complexity theory measures in the neural net setting
• GC captures the double-descent phenomenon: high-parameter-count networks can simultaneously fit training data and achieve low test error, contradicting classical overfitting intuition
• Standard training heuristics — overparameterized initialization, high learning rates, small batches, implicit gradient regularization, spectral norm, label noise, flatness regularization — all regulate GC via different mechanisms
• Research isolates individual heuristic effects using SGD without momentum, avoiding confounds; replicated with momentum and Adam with comparable results
• Tested on DNN and ResNet architectures using MNIST and CIFAR datasets
• Paper: "Why neural networks find simple solutions: the many regularizers of geometric complexity" (arXiv 2209.13083)

Connections: Google · Deepmind · Deep Learning · Regularization · Generalization

Source: https://www.marktechpost.com/2022/10/06/a-new-study-by-google-and-deepmind-introduces-geometric-complexity-gc-for-neural-network-analysis-and-understanding-of-deep-learning-models/