Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning

Pengyue Jia (City University of Hong Kong) · Xiangyu Zhao (Shanghai Jiao Tong University) · Wenlin Zhang (City University of Hong Kong) · Xiangyang Li (University of Science and Technology of China) · Kuicai Dong (Huawei International Pte. Ltd.) · Yichao Wang (Huawei Noah's Ark Lab) · Xiaopeng Li (National University of Defense Technology) · Yingyi Zhang (City University of Hong Kong) · Derong Xu (City University of Hong Kong) · Zhaocheng Du (Huawei Technologies Ltd.) · Huifeng Guo (Huawei Technologies Ltd.) · Ruiming Tang (Kuaishou- 快手科技)

adaptive workflowsanswer synthesisbenchmark datasetsdynamic retrievalevidence extractionexploration efficiencygradient conflictiterative context refinementoutcome-based reinforcement learningprocess-supervised reinforcement learningquery generationrag-proguideretrieval-augmented generationsparse reward signals

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by integrating external knowledge, yet traditional RAG systems struggle with static workflows and limited adaptability for complex, multistep reasoning tasks. Agentic RAG systems, such as DeepResearch, address these issues through dynamic retrieval, iterative context refinement, and adaptive workflows. However, recent methods like Search-R1, which rely on outcome-based reinforcement learning, face challenges such as low exploration efficiency, gradient conflict, and sparse reward signals. To tackle these limitations, we introduce ReasonRAG, a novel method that leverages RAG-ProGUIDE—a high-quality dataset providing fine-grained, process-level rewards for query generation, evidence extraction, and answer generation. By employing process-supervised reinforcement learning, ReasonRAG enhances LLMs’ autonomous capabilities in search, query generation, evidence extraction, and answer synthesis. Experimental results show that ReasonRAG, utilizing RAG-ProGUIDE, outperforms existing approaches like Search-R1 and traditional RAG systems, achieving superior performance on five benchmark datasets with only 5k training instances—significantly fewer than the 90k required by Search-R1. Our code is available at https://github.com/Applied-Machine-Learning-Lab/ReasonRAG.