PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning

Xinchen Zhang (Tsinghua University) · Zhong-Zhi Li (Institute of automation, Chinese academy of science, Chinese Academy of Sciences) · Yeyun Gong (Microsoft) · yelong shen (Microsoft) · Peijie Wang (Chinese Academy of Sciences) · Yujiu Yang (Tsinghua University) · Jie Wu (Tsinghua University) · Yizhen Zhang (Tsinghua University) · Yang Ding (Tsinghua University) · Shuoshuo Zhang (Microsoft) · Haoling Li (Tsinghua University) · Lei Ji (, Chinese Academy of Sciences)

exploration-exploitation trade-offlearned policieslearning efficiencymulti-image positional reasoningmultimodal reasoningoptimal behaviorspermutation of image sequencespositional relationshipsreinforcement learningresamplingrollout filtering mechanismspatial reasoning state-of-the-art performance task performance vision-language models

Inspired by the impressive reasoning capabilities demonstrated by reinforcement learning approaches like DeepSeek-R1, recent emerging research has begun exploring the use of reinforcement learning (RL) to enhance vision-language models (VLMs) for multimodal reasoning tasks. However, most existing multimodal reinforcement learning approaches remain limited to spatial reasoning within single-image contexts, yet still struggle to generalize to more complex and real-world scenarios involving multi-image positional reasoning, where understanding the relationships across images is crucial. To address this challenge, we propose a general reinforcement learning approach PeRL tailored for interleaved multimodal tasks, and a multi-stage strategy designed to enhance the exploration-exploitation trade-off, thereby improving learning efficiency and task performance. Specifically, we introduce permutation of image sequences to simulate varied positional relationships to explore more spatial and positional diversity. Furthermore, we design a rollout filtering mechanism for resampling to focus on trajectories that contribute most to learning optimal behaviors to exploit learned policies effectively. We evaluate our model on 5 widely-used multi-image benchmarks and 3 single-image benchmarks. Our experiments confirm that PeRL trained model consistently surpasses R1-related and interleaved VLM baselines by a large margin, achieving state-of-the-art performance on multi-image benchmarks, while preserving comparable performance on single-image tasks.