COS3D: Collaborative Open-Vocabulary 3D Segmentation

Pheng-Ann Heng (The Chinese University of Hong Kong) · Shi Qiu (Peking University) · Chi-Wing Fu (The Chinese University of Hong Kong) · Runsong Zhu (The Chinese University of Hong Kong) · Ka-Hei Hui (Autodesk) · Zhengzhe Liu (Carnegie Mellon University) · Qianyi Wu (Monash University) · Weiliang Tang (The Chinese University of Hong Kong)

adaptive language-to-instance prompt refinementclass-agnostic segmentationscollaborative fieldcollaborative prompt-segmentation frameworkgaussian-splattinghierarchical segmentationhigh-quality prompt-segmentationimage-based 3d segmentationinstance fieldinstance-to-language feature mappingintrinsic relationshiplanguage fieldopen-vocabulary 3d segmentationrobotics two-stage training strategy

Open-vocabulary 3D segmentation is a fundamental yet challenging task, requiring a mutual understanding of both segmentation and language. However, existing Gaussian-splatting-based methods rely either on a single 3D language field, leading to inferior segmentation, or on pre-computed class-agnostic segmentations, suffering from error accumulation. To address these limitations, we present COS3D, a new collaborative prompt-segmentation framework that contributes to effectively integrating complementary language and segmentation cues throughout its entire pipeline. We first introduce the new concept of collaborative field, comprising an instance field and a language field, as the cornerstone for collaboration. During training, to effectively construct the collaborative field, our key idea is to capture the intrinsic relationship between the instance field and language field, through a novel instance-to-language feature mapping and designing an efficient two-stage training strategy. During inference, to bridge distinct characteristics of the two fields, we further design an adaptive language-to-instance prompt refinement, promoting high-quality prompt-segmentation inference. Extensive experiments not only demonstrate COS3D's leading performance over existing methods on two widely-used benchmarks but also show its high potential to various applications,~\ie, novel image-based 3D segmentation, hierarchical segmentation, and robotics.