Recurrent Memory for Online Interdomain Gaussian Processes

Samir Bhatt (Imperial College London) · Wenlong Chen (Isomorphic Labs) · Yingzhen Li (Imperial College London) · Naoki Kiyohara (Imperial College London Canon Inc.) · Harrison Zhu (Copenhagen University) · Jacob Curran-Sebastian (Copenhagen University)

continual learningdiscriminative gp modelhigh-order polynomial projection operatorshippo frameworkinducing variablesinterdomain gp frameworkkernel matriceslong-term memoryode evolutiononline gaussian processorthogonal polynomial basis functionssequential datasparse gaussian process variational autoencodersvgp inducing pointstime-varying orthogonal projections

We propose a novel online Gaussian process (GP) model that is capable of capturing long-term memory in sequential data in an online learning setting. Our model, Online HiPPO Sparse Variational Gaussian Process (OHSVGP), leverages the HiPPO (High-order Polynomial Projection Operators) framework, which is popularized in the RNN domain due to its long-range memory modeling capabilities. We interpret the HiPPO time-varying orthogonal projections as inducing variables with time-dependent orthogonal polynomial basis functions, which allows the SVGP inducing points to memorize the process history. We show that the HiPPO framework fits naturally into the interdomain GP framework and demonstrate that the kernel matrices can also be updated online in a recurrence form based on the ODE evolution of HiPPO. We evaluate OHSVGP with online prediction for 1D time series, continual learning in discriminative GP model for data with multidimensional inputs, and deep generative modeling with sparse Gaussian process variational autoencoder, showing that it outperforms existing online GP methods in terms of predictive performance, long-term memory preservation, and computational efficiency.