Gradient Descent Models Are Kernel Machines (Deep Learning)

kernel-machinesgradient-descentdeep-learninginterpretability

Abstraction: Deep networks trained by gradient descent are kernel machines

Key points:

Pedro Domingos (arxiv:2012.00152) shows every model learned by gradient descent, regardless of architecture, is approximately equivalent to a kernel machine with a "path kernel"
The kernel measures similarity of two data points in the neighborhood of the parameter path taken during training — model weights are a superposition of training examples in gradient space
This reframes deep learning: nonlinearity is located in the kernel function; the superposition of kernels is entirely linear as long as loss is additive over data
Without regularity in the ground truth data-generating process, a neural net cannot generalize to points that deviate strongly (as defined by the kernel) from training examples
The result improves interpretability of deep network weights and should guide better learning algorithms
AGI implication: a sufficiently complex net can capture abstract similarities at multiple layers of representation, still encoded in the kernel

Connections: Pedro Domingos · Kernel Methods · Gradient Descent · Deep Learning