MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning

Jiangmiao Pang (Shanghai AI Laboratory ) · Zhen Luo (Shanghai Innovation Institute/Southern University of Science and Technology) · Feng Zheng (Southern University of Science and Technology) · Xudong XU (Shanghai AI Laboratory) · Zhaoyang Lyu (Shanghai AI Laboratory) · Ran Yi (Shanghai Jiao Tong University) · Weipeng Zhong (Shanghai Jiao Tong University) · Yichen Jin (Southeast University) · Jinkun Hao (Shanghai Jiaotong University) · Naifu Liang (University of California, San Diego) · Lizhuang Ma (Dept. of Computer Sci. & Eng., Shanghai Jiao Tong University)
baseline comparisondpo algorithmsinter-object relationsllm-based frameworkmanually crafted layoutsobject inferenceperformance evaluationphysically plausible scenesscene graph constructionspatial interrelation reasoningspatial reasoning chainsynthetic datasettabletop scenestask descriptionstask-oriented scene generation

The ability of robots to interpret human instructions and execute manipulation tasks necessitates the availability of task-relevant tabletop scenes for training. However, traditional methods for creating these scenes rely on time-consuming manual layout design or purely randomized layouts, which are limited in terms of plausibility or alignment with the tasks. In this paper, we formulate a novel task, namely task-oriented tabletop scene generation, which poses significant challenges due to the substantial gap between high-level task instructions and the tabletop scenes. To support research on such a challenging task, we introduce \textbf{MesaTask-10K}, a large-scale dataset comprising approximately 10,700 synthetic tabletop scenes with \emph{manually crafted layouts} that ensure realistic layouts and intricate inter-object relations. To bridge the gap between tasks and scenes, we propose a \textbf{Spatial Reasoning Chain} that decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction for the final 3D layout. We present \textbf{MesaTask}, an LLM-based framework that utilizes this reasoning chain and is further enhanced with DPO algorithms to generate physically plausible tabletop scenes that align well with given task descriptions. Exhaustive experiments demonstrate the superior performance of MesaTask compared to baselines in generating task-conforming tabletop scenes with realistic layouts.