Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs

Hao Fang (University of Washington) · Changle Zhou (University of Sydney, University of Sydney) · Jiawei Kong (Tsinghua University, Tsinghua University) · Kuofeng Gao (Tsinghua University) · Bin Chen (Harbin Institute of Technology, Shenzhen) · Tao Liang (University of Electronic Science and Technology of China) · Guojun Ma (Tencent Inc.) · Shu-Tao Xia (Tsinghua University)

bi-level optimizationcalibrated decoding strategyconditional pointwise mutual informationdecoding efficiencydecoding processgenerated responsehallucination mitigation hallucinations image tokens large vision-language models mutual dependencysampling text tokenstext token samplingtoken purification mechanismvisual and textual tokens

Large Vision-Language Models (LVLMs) are susceptible to hallucinations, where generated responses seem semantically plausible yet exhibit little or no relevance to the input image. Previous studies reveal that this issue primarily stems from LVLMs' over-reliance on language priors while disregarding the visual information during decoding. To alleviate this issue, we introduce a novel Conditional Pointwise Mutual Information (C-PMI) calibrated decoding strategy, which adaptively strengthens the mutual dependency between generated texts and input images to mitigate hallucinations. Unlike existing methods solely focusing on text token sampling, we propose to jointly model the contributions of visual and textual tokens to C-PMI, formulating hallucination mitigation as a bi-level optimization problem aimed at maximizing mutual information. To solve it, we design a token purification mechanism that dynamically regulates the decoding process by sampling text tokens remaining maximally relevant to the given image, while simultaneously refining image tokens most pertinent to the generated response. Extensive experiments across various benchmarks reveal that the proposed method significantly reduces hallucinations in LVLMs while preserving decoding efficiency.