The Rich and the Simple: On the Implicit Bias of Adam and SGD

Vatsal Sharan (University of Southern California (USC)) · Bhavya Vasudeva (USC) · Jung Lee (University of Southern California) · Mahdi Soltanolkotabi (University of Southern California)

bayes' optimal predictorbinary classification decision boundary distribution shifts first-order methodsgaussian datageneralization implicit bias optimization algorithmpopulation gradientssimplicity biasspurious correlations stochastic gradient descent test accuracy

Adam is the de facto optimization algorithm for several deep learning applications, but an understanding of its implicit bias and how it differs from other algorithms, particularly standard first-order methods such as (stochastic) gradient descent (GD), remains limited. In practice, neural networks (NNs) trained with SGD are known to exhibit simplicity bias