STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex

Ethan Trepka (Stanford University) · Ruobing Xia (Stanford University) · Shude Zhu (Stanford University) · Sharif Saleki (Stanford University) · Danielle Lopes (Stanford University) · Stephen Cital (TECH Global University) · Konstantin Willeke (Stanford University) · Mindy Kim (Brown University) · Tirin Moore (Stanford University)

benchmarking convolutional neural networksdorsal streamencoding modelslarge-scale datasetsnatural videosneural mechanismsneuronal responsesobject recognitionprimate visual systemrhesus macaquessingle neuron recordingssuperior temporal sulcusventral streamvisual input reconstruction

The primate visual system is typically divided into two streams — the ventral stream, responsible for object recognition, and the dorsal stream, responsible for encoding spatial relations and motion. Recent studies have shown that convolutional neural networks (CNNs) pretrained on object recognition tasks are remarkably effective at predicting neuronal responses in the ventral stream, shedding light on the neural mechanisms underlying object recognition. However, similar models of the dorsal stream remain underdeveloped due to the lack of large scale datasets encompassing dorsal stream areas. To address this gap, we present STSBench, a dataset of large-scale, single neuron recordings from over 2,000 neurons in the superior temporal sulcus (STS), a nearly 50-fold increase over existing dorsal stream datasets, collected while Rhesus macaques viewed thousands of unique, natural videos. We show that our dataset can be used for benchmarking encoding models of dorsal stream neuronal responses and reconstructing visual input from neural activity.