Fast Training of Large Kernel Models with Delayed Projections

Misha Belkin (University of California, San Diego) · Amirhesam Abedsoltan (University of California, San Diego) · Siyuan Ma (Google) · Parthe Pandit (Indian Institute of Technology, Bombay)

algorithm validationdata sizedelayed projectionseigenprokernel machineslarge datasets methodology model size performance optimizationpreconditioned stochastic gradient descentpsgdscalingtraining speedups

Classical kernel machines have historically faced significant challenges in scaling to large datasets and model sizes—a key ingredient that has driven the success of neural networks. In this paper, we present a new methodology for building kernel machines that can scale efficiently with both data size and model size. Our algorithm introduces delayed projections to Preconditioned Stochastic Gradient Descent (PSGD) allowing the training of much larger models than was previously feasible. We validate our algorithm, \EP4, across multiple datasets, demonstrating drastic training speedups without compromising the performance. Our implementation is publicly available at: https://github.com/EigenPro/EigenPro .