Deep Learning Through the Lens of Example Difficulty

Abstraction: Prediction depth as per-example measure of deep learning difficulty

Key points:

• Introduces "prediction depth" (effective prediction depth) as a measure of how computationally difficult a given input is for a model
• Prediction depth correlates with model uncertainty, confidence, accuracy, and speed of learning for each data point
• Difficult examples are categorized into three interpretable groups processed differently within deep networks
• Confirms that early layers generalize while later layers memorize; early layers also converge faster
• Networks learn easy data and simple functions first — consistent with frequency/simplicity bias literature
• Authors: Robert J. N. Baldock, Hartmut Maennel, Behnam Neyshabur (2021)

Connections: Arxiv · Deep Learning · Generalization · Memorization

Source: https://arxiv.org/abs/2106.09647