Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel

Wei Huang (RIKEN AIP) · Taiji Suzuki (The University of Tokyo/RIKEN-AIP) · Yilan Chen (University of California, San Diego) · Zhichao Wang (University of California, Berkeley) · Andi Han (RIKEN AIP) · Arya Mazumdar (University of California, San Diego)

generalization bound generalization guarantees gradient flow gradient-based optimizationkernel traceloss path kernelntkoptimization dynamics overparameterized neural networks rademacher complexityrkhs normstability analysisstatic kernelstraining loss gradientstraining trajectoryuniform convergence

Gradient-based optimization methods have shown remarkable empirical success, yet their theoretical generalization properties remain only partially understood. In this paper, we establish a generalization bound for gradient flow that aligns with the classical Rademacher complexity bounds for kernel methods—specifically those based on the RKHS norm and kernel trace—through a data-dependent kernel called the loss path kernel (LPK). Unlike static kernels such as NTK, the LPK captures the entire training trajectory, adapting to both data and optimization dynamics, leading to tighter and more informative generalization guarantees. Moreover, the bound highlights how the norm of the training loss gradients along the optimization trajectory influences the final generalization performance. The key technical ingredients in our proof combine stability analysis of gradient flow with uniform convergence via Rademacher complexity. Our bound recovers existing kernel regression bounds for overparameterized neural networks and shows the feature learning capability of neural networks compared to kernel methods. Numerical experiments on real-world datasets validate that our bounds correlate well with the true generalization gap.