Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos

Ashkan Mirzaei (NVIDIA) · Antonio Torralba (MIT) · Sanja Fidler (University of Ljubljana) · Ziwei Liu (Nanyang Technological University) · Jiawei Ren (NVIDIA) · Igor Gilitschenski (University of Toronto) · Zan Gojcic (NVIDIA) · hanxue liang · Cengiz Oztireli (University of Cambridge & Google) · Huan Ling (NVIDIA) · Jiahui Huang (NVIDIA)

3d gaussian splattingbullet-time scenecontext framesdynamic contentdynamic scene datasetsfeed-forward modelgeneralizabilitymonocular dynamic videomotion-aware modelnovel view synthesis optimization-based approachesreal-time reconstructionscalability scene reconstruction state-of-the-art performance

Recent advancements in static feed-forward scene reconstruction have demonstrated significant progress in high-quality novel view synthesis. However, these models often struggle with generalizability across diverse environments and fail to effectively handle dynamic content. We present BTimer (short for Bullet Timer), the first motion-aware feed-forward model for real-time reconstruction and novel view synthesis of dynamic scenes. Our approach reconstructs the full scene in a 3D Gaussian Splatting representation at a given target (‘bullet’) timestamp by aggregating information from all the context frames. Such a formulation allows BTimer to gain scalability and generalization by leveraging both static and dynamic scene datasets. Given a casual monocular dynamic video, BTimer reconstructs a bullet-time scene within 150ms while reaching state-of-the-art performance on both static and dynamic scene datasets, even compared with optimization-based approaches.