Why does Deep Learning work?

Abstraction: Deep learning energy landscape as spin funnel not spin glass

Key points:

• LeCun (2015) analyzed deep network training through the lens of spin-glass theory; loss surfaces of multilayer networks are studied via SGD behavior
• Author's conjecture: deep learning systems are "spin funnels" (energy landscapes of minimal frustration, roughly convex funnel shape) rather than spin glasses (highly non-convex with many local minima)
• Analogy from protein folding: Wolynes' spin funnel model resolves Levinthal's paradox — as network size increases, the funnel gets sharper
• Key implication: SGD works not because the landscape is convex globally, but because it reaches the top of a funnel and then descends easily; the real challenge is avoiding overtraining
• 2018 paper "Visualizing the Loss Landscape of Neural Nets" confirmed: deep networks transition from nearly-convex to chaotic loss landscapes; skip connections promote flat minima and prevent this transition
• Unsupervised deep learning conjectured to relate to the variational renormalization group

Connections: Yann LeCun · Deep Learning · Neural Networks · Energy Landscape · Optimization

Source: https://charlesmartin14.wordpress.com/2015/03/25/why-does-deep-learning-work/