V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation

Boxin Shi (Peking University) · Yi Wang (Beijing University of Posts and Telecommunications) · Minggui Teng · Hanyue Lou (Peking University) · Jinxiu Liang (National Institute of Informatics)

diverse videosdynamic rangeevent datasetsevent vision modelsevent-based camerasmodel robustness optical flow estimationparameter randomizationstorage requirementssynthetic data generation temporal resolution training datasetsvideo reconstructionvideo-to-voxelvoxel grid representations

Event-based cameras offer unique advantages such as high temporal resolution, high dynamic range, and low power consumption. However, the massive storage requirements and I/O burdens of existing synthetic data generation pipelines and the scarcity of real data prevent event-based training datasets from scaling up, limiting the development and generalization capabilities of event vision models. To address this challenge, we introduce Video-to-Voxel (V2V), an approach that directly converts conventional video frames into event-based voxel grid representations, bypassing the storage-intensive event stream generation entirely. V2V enables a 150× reduction in storage requirements while supporting on-the-fly parameter randomization for enhanced model robustness. Leveraging this efficiency, we train several video reconstruction and optical flow estimation model architectures on 10,000 diverse videos totaling 52 hours—an order of magnitude larger than existing event datasets, yielding substantial improvements.