A New Link to an Old Model Could Crack the Mystery of Deep Learning | Quanta Magazine

Abstraction: Infinite-width neural networks mathematically equivalent to kernel machines

Key points:

• Over-parameterized deep networks (billions of params) generalize well despite classical theory predicting overfitting—the mystery this work addresses.
• Radford Neal (1994) showed a single-hidden-layer infinite-width network equals a Gaussian process at initialization; extended to deep networks in 2017.
• Arthur Jacot (NeurIPS 2018) showed infinite-width deep nets remain equivalent to a fixed kernel (neural tangent kernel, NTK) throughout training, not just at initialization.
• NTK equivalence proves gradient descent converges to a global minimum in the idealized case—explaining why these nets don't get stuck.
• Kernel methods (e.g., SVMs) dominated ML before 2012's AlexNet; the new link suggests their theory still illuminates modern deep learning.
• Key limitation: practical networks have finite width and weights can change dramatically during training, so the equivalence may not fully transfer.

Connections: Google Brain · Deep Learning · Kernel Methods · Neural Tangent Kernel

Source: https://www.quantamagazine.org/a-new-link-to-an-old-model-could-crack-the-mystery-of-deep-learning-20211011/