SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning

Yu Zhang (HKUST) · Zhongwei Wan (Ohio State University, Columbus) · Yangfan He · Shen Yan (ByteDance Seed) · Jing Xiong (University of Hong Kong) · Mi Zhang (The Ohio State University) · Zhihao Dou (Georgia Institute of Technology) · Che Liu (Imperial College London) · Dongfei Cui (Duke University) · Qinjian Zhao (Kean University) · Hui Shen (University of Michigan - Ann Arbor) · Yi Xin (Nanjing University) · Yifan Jiang (USC, ISI) · Chaofan Tao (University of Hong Kong)

cognitive reflectiongroup relative policy optimization high-quality dataset multimodal large language modelsmultimodal reasoning benchmarksqwen-2.5-vl-32bqwen-2.5-vl-7breasoning accuracy reasoning tasksreflection qualityreflection-awarereinforcement learning reward mechanism self-correction self-reflection

Multimodal large language models (MLLMs) have shown promising capabilities in reasoning tasks, yet still struggle significantly with complex problems requiring explicit self-reflection and self-correction, especially compared to their unimodal text-based counterparts. Existing reflection methods are simplistic and struggle to generate meaningful, instructive feedback, as the reasoning ability and knowledge limits of pre-trained models are largely fixed during initial training. To overcome these challenges, we propose \textit{multimodal \textbf{S}elf-\textbf{R}eflection enhanced reasoning with Group Relative \textbf{P}olicy \textbf{O}ptimization} \textbf{SRPO}, a two-stage reflection-aware reinforcement learning (RL) framework explicitly designed to enhance multimodal LLM reasoning. In the first stage, we construct a high-quality, reflection-focused dataset under the guidance of an advanced MLLM, which generates reflections based on initial responses to help the policy model to learn both reasoning and self-reflection. In the second stage, we introduce a novel reward mechanism within the GRPO framework that encourages concise and cognitively meaningful reflection while avoiding redundancy. Extensive experiments across multiple multimodal reasoning benchmarks—including MathVista, MathVision, Mathverse, and MMMU-Pro—using Qwen-2.5-VL-7B and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms state-of-the-art models, achieving notable improvements in both reasoning accuracy and reflection quality.