Efficient Training of Minimal and Maximal Low-Rank Recurrent Neural Networks

Low-rank recurrent neural networks (RNNs) provide a powerful framework for characterizing how neural systems solve complex cognitive tasks. However, fitting and interpreting these networks remains an important open problem. In this paper, we develop new methods for efficiently fitting low-rank RNNs in ''teacher-training'' settings. In particular, we build upon the neural engineering framework (NEF), in which RNNs are viewed as approximating an ordinary differential equation (ODE) of interest using a set of random nonlinear basis functions. This view provides geometric insight into how the choice of neural nonlinearity (e.g. tanh, ReLU) and the distribution of model parameters affects an RNN's representational capacity. We adapt this framework for online training and demonstrate better performance with significantly smaller networks compared to FORCE. Additionally, we outperform backpropagation-trained networks of similar size, while requiring substantially less training time. Next, we ask: how many neurons