Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs

Shanghang Zhang (Peking University) · Qizhe Zhang (Peking University) · Mengzhen Liu (Peking University) · Lichen Li (Peking University) · Ming Lu (Nanjing University) · Yuan Zhang (Ohio State University, Columbus) · Junwen Pan (Bytedance) · Qi She (Bytedance AI Lab)

attention-based pruningconditional diversityconditional similaritycuda latencydeterminantal point processflops reduction input representationinstruction relevancemodel-agnostic multimodal large language modelssimilarity-based pruningstate-of-the-art performance training-freeuser instructionsvision-language benchmarks visual token pruning

In multimodal large language models (MLLMs), the length of input visual tokens is often significantly greater than that of their textual counterparts, leading to a high inference cost. Many works aim to address this issue by removing redundant visual tokens. However, current approaches either rely on attention-based pruning, which retains numerous duplicate tokens, or use similarity-based pruning, overlooking the instruction relevance, consequently causing suboptimal performance. In this paper, we go beyond attention or similarity by proposing a novel visual token pruning method named **CDPruner**, which maximizes the conditional diversity of retained tokens. We first define the conditional similarity between visual tokens conditioned on the instruction, and then reformulate the token pruning problem with determinantal point process (DPP) to maximize the conditional diversity of the selected subset. The proposed CDPruner is training-free and model-agnostic, allowing easy application to various MLLMs. Extensive experiments across diverse MLLMs show that CDPruner establishes new state-of-the-art on various vision-language benchmarks. By maximizing conditional diversity through DPP, the selected subset better represents the input images while closely adhering to user instructions, thereby preserving strong performance even with high reduction ratios. When applied to LLaVA, CDPruner reduces FLOPs by **95\%** and CUDA latency by **78\%**, while maintaining **94\%** of the original accuracy. Our code is available at https://github.com/Theia-4869/CDPruner.