Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning

An Zhang (University of Science and Technology of China) · Junfeng Fang (University of Science and Technology of China) · Xiang Wang (National University of Singapore) · Yaorui Shi (University of Science and Technology of China) · Sihang Li (University of Science and Technology of China) · Chang Wu (University of Science and Technology of China) · ZHIYUAN LIU (National University of Singapore) · Hengxing Cai (SUN YAT-SEN UNIVERSITY)

answer generation complex reasoningevidence distillationevidence synthesisgroup relative policy optimizationiterative filteringknowledge refinementmulti-hop qa post-training framework reinforcement learningretrieval-augmented reasoningretrieval-specific rewardssearch-and-refine paradigmsingle-hop qa

Large language models have demonstrated impressive reasoning capabilities but are inherently limited by their knowledge reservoir. Retrieval-augmented reasoning mitigates this limitation by allowing LLMs to query external resources, but existing methods often retrieve irrelevant or noisy information, hindering accurate reasoning. In this paper, we propose **AutoRefine**, a reinforcement learning post-training framework that adopts a new "search-and-refine-during-think" paradigm. AutoRefine introduces explicit knowledge refinement steps between successive search calls, enabling the model to iteratively filter, distill, and organize evidence before generating an answer. Furthermore, we incorporate tailored retrieval-specific rewards alongside answer correctness rewards using group relative policy optimization. Experiments on single-hop and multi-hop QA benchmarks demonstrate that AutoRefine significantly outperforms existing approaches, particularly in complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine issues frequent, higher-quality searches and synthesizes evidence effectively.