ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding

Jun Zhu (Tsinghua University) · junliang ye (Tsinghua University) · Zhengyi Wang (Tsinghua University, Tsinghua University) · Ruowen Zhao (Tsinghua University) · Shenghao Xie (Peking University)

3d content3d generation capabilities3d vector-quantized variational autoencoder3d-alpaca3d-aware discrete tokens3d-native aicontinuous training datasetdiscrete latent spacegpt-4oinstruction-based fine-tuningmultimodal large language modelsnative 3d understandingqwen-2.5-vl-7b-instructshape reconstructionshape representationshapellm-omni

Recently, the powerful text-to-image capabilities of GPT-4o have led to growing appreciation for native multimodal large language models. However, its multimodal capabilities remain confined to images and text. Yet beyond images, the ability to understand and generate 3D content is equally crucial. To address this gap, we propose ShapeLLM-Omni—a native 3D large language model capable of understanding and generating 3D assets and text in any sequence. First, we train a 3D vector-quantized variational autoencoder (VQVAE), which maps 3D objects into a discrete latent space to achieve efficient and accurate shape representation and reconstruction. Building upon the 3D-aware discrete tokens, we innovatively construct a large-scale continuous training dataset named 3D-Alpaca, encompassing generation, comprehension, and editing, thus providing rich resources for future research and training. Finally, we perform instruction-based fine-tuning of the Qwen-2.5-vl-7B-Instruct model on the 3D-Alpaca dataset, equipping it with native 3D understanding and generation capabilities. Our work represents an effective step toward extending multimodal large language models with fundamental 3D intelligence, paving the way for future advances in 3D-native AI.