InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactions

Jun Wang (iWudao Tech) · Zenglin Xu (Fudan University) · Yong Dai (Tencent AI Lab) · Liangjian Wen ( Southwestern University of Finance and Economics) · Qun Dai (Southwest University of Finance and Economics) · Jianzhuang Liu (Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences) · Jiangtao Zheng (Southwest University of Finance and Economics) · Dongkai Wang (Southwest University of Finance and Economics) · Zhao Kang (University of Electronic Science and Technology of China) · Jiang Duan (Southwest University of Finance and Economics)

benchmarkscomprehensive synergistic informationcomputationally prohibitivecontrastive information extractioninfinite masking strategyinfmasking losslarge-scale real-world datasetsmasked representationsmultimodal representation learningmutual information maximizationpartial modality combinationsricher interactionsstate-of-the-art performancestochastically occludessynergistic interactionsunmasked fused representations

In multimodal representation learning, synergistic interactions between modalities not only provide complementary information but also create unique outcomes through specific interaction patterns that no single modality could achieve alone. Existing methods may struggle to effectively capture the full spectrum of synergistic information, leading to suboptimal performance in tasks where such interactions are critical. This is particularly problematic because synergistic information constitutes the fundamental value proposition of multimodal representation. To address this challenge, we introduce InfMasking, a contrastive synergistic information extraction method designed to enhance synergistic information through an Infinite Masking strategy. InfMasking stochastically occludes most features from each modality during fusion, preserving only partial information to create representations with varied synergistic patterns. Unmasked fused representations are then aligned with masked ones through mutual information maximization to encode comprehensive synergistic information. This infinite masking strategy enables capturing richer interactions by exposing the model to diverse partial modality combinations during training. As computing mutual information estimates with infinite masking is computationally prohibitive, we derive an InfMasking loss to approximate this calculation. Through controlled experiments, we demonstrate that InfMasking effectively enhances synergistic information between modalities. In evaluations on large-scale real-world datasets, InfMasking achieves state-of-the-art performance across seven benchmarks. Code is released at https://github.com/brightest66/InfMasking.