What Can RL Bring to VLA Generalization? An Empirical Study

Yu Wang (Tsinghua University) · Xinlei Chen (Tsinghua University) · Feng Gao (IIIS, Tsinghua University) · YI WU (Tsinghua University) · Jijia Liu (Tsinghua University) · Bingwen Wei (Tsinghua University) · Qingmin Liao (Tsinghua University, Tsinghua University) · Chao Yu (Tsinghua University, Tsinghua University)

distribution shifts dpo efficient training embodied aiexecution robustnessgeneralization grpollm-derived methodsppo reinforcement learning semantic understanding supervised fine-tuningtrial-and-errorvision-language actionvla generalization

Large Vision-Language Action (VLA) models have shown significant potential for embodied AI. However, their predominant training via supervised fine-tuning (SFT) limits generalization due to susceptibility to compounding errors under distribution shifts. Reinforcement learning (RL) offers a path to overcome these limitations by optimizing for task objectives via trial-and-error, yet a systematic understanding of its specific generalization benefits for VLAs compared to SFT is lacking. To address this, our study introduces a comprehensive benchmark for evaluating VLA generalization and systematically investigates the impact of RL fine-tuning across diverse visual, semantic, and execution dimensions. Our extensive experiments reveal that RL fine-tuning, particularly with PPO, significantly enhances generalization in semantic understanding and execution robustness over SFT, while maintaining comparable visual robustness. We identify PPO as a more effective RL algorithm for VLAs than LLM-derived methods like DPO and GRPO. We also develop a simple recipe for efficient PPO training on VLAs, and demonstrate its practical utility for improving VLA generalization. The project page is at https://rlvla.github.io