Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime
continual learningconvergence analysisexcess riskinterpolation regimeleast squares regressionnear optimal rateoptimization techniquesover-parameterized modelspopulation convergencerandomized kaczmarz methodrealizable linear regressionsmooth convex objectivesstepsizestochastic gradient descentvariance of stochastic gradients
We study population convergence guarantees of stochastic gradient descent (SGD) for smooth convex objectives in the interpolation regime, where the noise at optimum is zero or near zero. The behavior of the last iterate of SGD in this setting