Martian World Model: Controllable Video Synthesis with Physically Accurate 3D Reconstructions

Yao Zhao (Beijing Jiaotong University) · Panwang Pan (ByteDance) · Zhangyang "Atlas" Wang (University of Texas at Austin) · Yunchao Wei (Beijing Jiaotong University) · Yue Wang (Soochow University, China) · Longfei Li (Beijing Jiaotong University) · Zhiwen Fan (University of Texas, Austin) · Wenyan Cong (University of Texas at Austin) · Xinhang Liu (HKUST) · Yuyang Yin (Beijing Jiaotong University) · Matt Foutter (Stanford University) · Chenyu You (State University of New York at Stony Brook) · Marco Pavone (Stanford University)
3d-consistent framescamera trajectoriesdata curation frameworkdomain gaphigh-fidelity 3d videomarsgenmartian landscape synthesismetric-scale resolutionmultimodal conditioning datamultimodal mars synthesisnasa planetary data systemphysics-accurate modelsstereo navigation imagestextual promptsvideo-based terrain generator

The synthesis of realistic Martian landscape videos, essential for mission rehearsal and robotic simulation, presents unique challenges. These primarily stem from the scarcity of high-quality Martian data and the significant domain gap relative to terrestrial imagery.To address these challenges, we introduce a holistic solution comprising two main components: 1) a data curation framework, Multimodal Mars Synthesis (M3arsSynth), which processes stereo navigation images to render high-fidelity 3D video sequences. 2) a video-based Martian terrain generator (MarsGen), that utilizes multimodal conditioning data to accurately synthesize novel, 3D-consistent frames. Our data are sourced from NASA’s Planetary Data System (PDS), covering diverse Martian terrains and dates, enabling the production of physics-accurate 3D surface models at metric-scale resolution. During inference, MarsGen is conditioned on an initial image frame and can be guided by specified camera trajectories or textual prompts to generate new environments.Experimental results demonstrate that our solution surpasses video synthesis approaches trained on terrestrial data, achieving superior visual quality and 3D structural consistency.