Boosting Knowledge Utilization in Multimodal Large Language Models via Adaptive Logits Fusion and Attention Reallocation

Shijian Lu (Nanyang Technological University) · Wenbin An (Xi'an Jiaotong University) · Jiahao Nie (Nanyang Technological University) · Feng Tian (Xi'an Jiaotong University) · Haonan Lin (Xi‘an Jiaotong University) · mingxiang cai (Beijing Jiaotong University) · Yaqiang Wu (Lenovo Research) · QianYing Wang (lenovo group) · Xiaoqin Zhang (Wenzhou University)

adaptive logits fusionattention biasattention reallocationcontextual knowledgeexternal databasesknowledge conflicts knowledge-intensive tasks multimodal large language models output logits parametric knowledge performance benchmarksplug-and-play approachquery-context relevanceretrieval augmented generation

Despite their recent progress, Multimodal Large Language Models (MLLMs) often struggle in knowledge-intensive tasks due to the limited and outdated parametric knowledge acquired during training. Multimodal Retrieval Augmented Generation addresses this issue by retrieving contextual knowledge from external databases, thereby enhancing MLLMs with expanded knowledge sources. However, existing MLLMs often fail to fully leverage the retrieved contextual knowledge for response generation. We examine representative MLLMs and identify two major causes, namely, attention bias toward different tokens and knowledge conflicts between parametric and contextual knowledge. To this end, we design Adaptive Logits Fusion and Attention Reallocation (ALFAR), a training-free and plug-and-play approach that improves MLLM responses by maximizing the utility of the retrieved knowledge. Specifically, ALFAR tackles the challenges from two perspectives. First, it alleviates attention bias by adaptively shifting attention from visual tokens to relevant context tokens according to query-context relevance. Second, it decouples and weights parametric and contextual knowledge at output logits, mitigating conflicts between the two types of knowledge. As a plug-and-play method, ALFAR achieves superior performance across diverse datasets without requiring additional training or external tools. Extensive experiments over multiple MLLMs and benchmarks show that ALFAR consistently outperforms the state-of-the-art by large margins. Our code and data are available at https://github.com/Lackel/ALFAR.