4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time

Kai Zhang (The Ohio State University) · Ziqiao Ma (University of Michigan) · Xuweiyi Chen (University of Virginia) · Shoubin Yu (Department of Computer Science, University of North Carolina at Chapel Hill) · Sai Bi (Adobe Systems) · Ziwen Chen (Oregon State University) · Sihan Xu (University of Michigan - Ann Arbor) · Jianing Yang (University of Michigan) · Zexiang Xu (Adobe Research) · Kalyan Sunkavalli (Adobe Research) · Mohit Bansal (UNC Chapel Hill) · Joyce Chai (University of Michigan) · Hao Tan (Adobe Systems)
4d pretraining4d reconstruction modelaccurate 4d reconstructiondiverse camera setupsefficient renderinggenerative modelsgeometry-based methodsnovel object generalizationnovel view-time combinationsoptimization-based approachesper-pixel 4d gaussian primitivesspace-time representationsspatiotemporal pretrainingtime interpolationunified space-time representation

Can we scale 4D pretraining to learn general space-time representations that reconstruct an object from a few views at some times to any view at any time? We provide an affirmative answer with 4D-LRM, the first large-scale 4D reconstruction model that takes input from unconstrained views and timestamps and renders arbitrary novel view-time combinations. Unlike prior 4D approaches, e.g., optimization-based, geometry-based, or generative, that struggle with efficiency, generalization, or faithfulness, 4D-LRM learns a unified space-time representation and directly predicts per-pixel 4D Gaussian primitives from posed image tokens across time, enabling fast, high-quality rendering at, in principle, infinite frame rate. Our results demonstrate that scaling spatiotemporal pretraining enables accurate and efficient 4D reconstruction. We show that 4D-LRM generalizes to novel objects, interpolates across time, and handles diverse camera setups. It reconstructs 24-frame sequences in one forward pass with less than 1.5 seconds on a single A100 GPU.