NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation

Tianyu Pang (Sea AI Lab) · Chao Du (Sea AI Lab) · Michael Qizhe Shieh (National University of Singapore) · Haonan Wang (The Hong Kong University of Science and Technology) · Xiangyan Liu (National University of Singapore) · Jinjie Ni (National University of Singapore) · Zijian Wu (National University of Singapore) · Longxu Dou (Sea AI Lab)

data augmentationdistortion strengthinductive biases model generalizabilitynoise annealingnoisyrolloutout-of-domain reasoningpolicy explorationreasoning behaviorsreinforcement learning test-time compute training stability trajectories vision-language models visual perception

Recent advances in reinforcement learning (RL) have strengthened the reasoning capabilities of vision-language models (VLMs). However, enhancing policy exploration to better scale test-time compute remains largely underexplored. In addition, VLMs continue to struggle with imperfect visual perception, which in turn affects the subsequent reasoning process. To this end, we propose **NoisyRollout**, a simple yet effective data augmentation method that mixes trajectories from both clean and moderately distorted images during RL training. By injecting targeted diversity in visual perception and the resulting reasoning patterns, NoisyRollout promotes better policy exploration through vision-oriented inductive biases, ultimately leading to more robust reasoning behaviors. We further adopt a noise annealing schedule that gradually reduces distortion strength over training, leveraging noisy signals early on while ensuring training stability in later stages. Crucially, our method is easy-to-adopt—**requiring no additional training cost and no modifications to the RL objective**. Extensive experiments on $2$ distinct training datasets demonstrate that NoisyRollout achieves state-of-the-art performance among open-source RL-tuned models across $5$ out-of-domain reasoning and perception benchmarks. Furthermore, we validate the effectiveness of NoisyRollout across model sizes ($7$B and $32$B) and data scales (from $1$K to $6$K), highlighting its generalizability and scalability.