Visualizing the Loss Landscape of Neural Nets

Abstraction: Filter normalization method reveals how architecture shapes neural loss geometry

Key points:

• Introduces "filter normalization" to enable meaningful side-by-side comparisons of loss landscapes across different network architectures
• Skip connections (e.g., ResNets) produce significantly smoother, less chaotic loss landscapes that are easier to train
• Training hyperparameters (batch size, learning rate, optimizer choice) affect the shape of minimizers found, not just final performance
• Broader, flatter minimizers tend to generalize better than sharp narrow ones
• Loss landscape visualization explains empirically observed differences in trainability between architectures
• arXiv:1712.09913; authors: Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein (2017)

Connections: Arxiv · Loss Landscape · Neural Network Optimization · Skip Connections

Source: https://arxiv.org/abs/1712.09913