SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning

Jiaqi Huang (Tsinghua University, Tsinghua University) · Zunnan Xu (Tsinghua University) · Jun Zhou (Ant Financial) · Ting Liu (Harbin Institute of Technology) · Yicheng Xiao (Southern University of Science and Technology) · Mingwen Ou (Tsinghua University, Tsinghua University) · Bowen Ji (Tsinghua University, Tsinghua University) · Xiu Li (Bytedance) · Kehong Yuan (Tsinghua University, Tsinghua University)

fine-grained reasoning image segmentationimage understanding tasksmanually annotated datasetsmultimodal large modelsoptimization objective performance benchmarksreasoning alignmentreasoning processes reinforcement learning segment anything modelsegmentation settingssegmentation-oriented reasoning capabilitiestask-specific rewards training samples

Leveraging multimodal large models for image segmentation has become a prominent research direction. However, existing approaches typically rely heavily on manually annotated datasets that include explicit reasoning processes, which are costly and time-consuming to produce. Recent advances suggest that reinforcement learning (RL) can endow large models with reasoning capabilities without requiring such reasoning-annotated data. In this paper, we propose SAM-R1, a novel framework that enables multimodal large models to perform fine-grained reasoning in image understanding tasks. Our approach is the first to incorporate fine-grained segmentation settings during the training of multimodal reasoning models. By integrating task-specific, fine-grained rewards with a tailored optimization objective, we further enhance the model's reasoning and segmentation alignment. We also leverage the Segment Anything Model (SAM) as a strong and flexible reward provider to guide the learning process. With only 3k training samples, SAM-R1 achieves strong performance across multiple benchmarks, demonstrating the effectiveness of reinforcement learning in equipping multimodal models with segmentation-oriented reasoning capabilities.