OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation

Ming-Ming Cheng (Nankai University) · Qibin Hou (Nankai University) · Bo-Wen Yin (Nankai University) · Jiao-Long Cao (Nankai University) · Xuying Zhang (Nankai University) · Yuming Chen (Nankai University)

efficient pretrainingkitti-360large-scale datasetmodality information encodingmulti-modal cluesmulti-modal datasetsmulti-modal learning frameworknyu depthv2omnisegmentorperceptual capabilitiespretrain-and-finetune pipelinerepresentation learning semantic segmentationstate-of-the-art recordsuniversal multi-modal pretraining

Recent research on representation learning has proved the merits of multi-modal clues for robust semantic segmentation. Nevertheless, a flexible pretrain-and-finetune pipeline for multiple visual modalities remains unexplored. In this paper, we propose a novel multi-modal learning framework, termed OmniSegmentor. It has two key innovations: 1) Based on ImageNet, we assemble a large-scale dataset for multi-modal pretraining, called OmniSegmentor, which contains five popular visual modalities; 2) We provide an efficient pretraining manner to endow the model with the capacity to encode different modality information in the OmniSegmentor. For the first time, we introduce a universal multi-modal pretraining framework that consistently amplifies the model's perceptual capabilities across various scenarios, regardless of the arbitrary combination of the involved modalities. Remarkably, our OmniSegmentor achieves new state-of-the-art records on a wide range of multi-modal semantic segmentation datasets, including NYU Depthv2, EventScape, MFNet, DeLiVER, SUNRGBD, and KITTI-360. Data, model checkpoints, and source code will be made publicly available: https://github.com/VCIP-RGBD/DFormer.