Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration

Lianhui Qin (University of California, San Diego) · Ziqiao Ma (University of Michigan) · Xiaokang Ye (University of California, San Diego) · Jiawei Ren (NVIDIA) · Yan Zhuang (University Of Science And Technology Of China) · Xuhong He (CMU, Carnegie Mellon University) · Tianmin Shu (Johns Hopkins University) · Zhiting Hu (University of California, San Diego) · Jianzhi Shen (Johns Hopkins University) · Ruixuan Zhang (Johns Hopkins University) · Tianai Yue (Johns Hopkins University) · Muhammad Faayez (Johns Hopkins University) · Xiyan Zhang (Johns Hopkins University)
3d spatial reasoningembodied aifoundation modelsgeneralist roboticsgrounded communicationinstructions groundinglong-range navigationmulti-agent search taskmulti-robot collaborationmulti-robot controlmultimodal inputsmultimodal instruction-followingphotorealistic urban environmentsprocedural generationunreal engine 5

Recent advances in foundation models have shown promising results in developing generalist robotics that can perform diverse tasks in open-ended scenarios given multimodal inputs. However, current work has been mainly focused on indoor, household scenarios. In this work, we present SimWorld-Robotics (SWR), a simulation platform for embodied AI in large-scale, photorealistic urban environments. Built on Unreal Engine 5, SWR procedurally generates unlimited photorealistic urban scenes populated with dynamic elements such as pedestrians and traffic systems, surpassing prior urban simulations in realism, complexity, and scalability. It also supports multi-robot control and communication. With these key features, we build two challenging robot benchmarks: (1) a multimodal instruction-following task, where a robot must follow vision-language navigation instructions to reach a destination in the presence of pedestrians and traffic; and (2) a multi-agent search task, where two robots must communicate to cooperatively locate and meet each other. Unlike existing benchmarks, these two new benchmarks comprehensively evaluate a wide range of critical robot capacities in realistic scenarios, including (1) multimodal instructions grounding, (2) 3D spatial reasoning in large environments, (3) safe, long-range navigation with people and traffic, (4) multi-robot collaboration, and (5) grounded communication. Our experimental results demonstrate that state-of-the-art models, including vision-language models (VLMs), struggle with our tasks, lacking robust perception, reasoning, and planning abilities necessary for urban environments.