Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval

Jian Xiao (Hefei University of Technology) · Zijie Song (Anhui University) · Jialong Hu (Hefei University of Technology) · Hao Cheng (Hefei University of Technology) · Jia Li (The Hong Kong University of Science and Technology) · Zhenzhen Hu (Hefei University of Technology) · Richang Hong (Hefei University of Technology)

alignment accuracyanchor representationscontrastive learninggap-aware retrievalinfonce lossmodality gapmultivariate first-order taylor expansionnoisy hard negativesoptimization tensionpair-specific incrementrelaxed compressionrobustnessstructure-aware correctiontrust-region constraintvariational information bottleneck

Recent progress in text–video retrieval has been largely driven by contrastive learning. However, existing methods often overlook the effect of the modality gap, which causes anchor representations to undergo in-place optimization (i.e., optimization tension) that limits their alignment capacity. Moreover, noisy hard negatives further distort the semantics of anchors. To address these issues, we propose GARE, a Gap-Aware Retrieval framework that introduces a learnable, pair-specific increment $\Delta_{ij}$ between text $t_i$ and video $v_j$, redistributing gradients to relieve optimization tension and absorb noise. We derive $\Delta_{ij}$ via a multivariate first-order Taylor expansion of the InfoNCE loss under a trust-region constraint, showing that it guides updates along locally consistent descent directions. A lightweight neural module conditioned on the semantic gap couples increments across batches for structure-aware correction. Furthermore, we regularize $\Delta$ through a variational information bottleneck with relaxed compression, enhancing stability and semantic consistency. Experiments on four benchmarks demonstrate that GARE consistently improves alignment accuracy and robustness, validating the effectiveness of gap-aware tension mitigation.