UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation

Shijian Lu (Nanyang Technological University) · Youwei Pang (Dalian University of Technology) · Xiaoqi Zhao (Yale University) · Lihe Zhang (Dalian University of Technology) · Huchuan Lu (Dalian University of Technology) · Georges Fakhri (Yale University) · Xiaofeng Liu (University of Michigan - Ann Arbor) · Chenyang Yu (Dalian University of Technology)

cross-modal fusiondeployment challengesfeature space distancefrozen encoderhierarchical self-supervised compensationhybrid consistency constrainthybrid shuffled-masking augmentationlightweight reverse attention adaptermodality reconstructionmodality-invariant contrastive learningmri-based brain tumor segmentationmulti-modal image segmentationperceptual semanticsrgb-d semantic segmentationrgb-d/t salient object segmentationstate-of-the-art methods

Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance. While existing methods address training-inference modality gaps via specialized per-combination models, they introduce high deployment costs by requiring exhaustive model subsets and model-modality matching. In this work, we propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC). Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels. First, we adopt modality reconstruction with the hybrid shuffled-masking augmentation, encouraging the model to learn the intrinsic modality characteristics and generate meaningful representations for missing modalities through cross-modal fusion. Next, modality-invariant contrastive learning implicitly compensates the feature space distance among incomplete-complete modality pairs. Furthermore, the proposed lightweight reverse attention adapter explicitly compensates for the weak perceptual semantics in the frozen encoder. Last, UniMRSeg is fine-tuned under the hybrid consistency constraint to ensure stable prediction under all modality combinations without large performance fluctuations. Without bells and whistles, UniMRSeg significantly outperforms the state-of-the-art methods under diverse missing modality scenarios on MRI-based brain tumor segmentation, RGB-D semantic segmentation, RGB-D/T salient object segmentation. The code will be released at \url{https://github.com/Xiaoqi-Zhao-DLUT/UniMRSeg}.