TTRL: Test-Time Reinforcement Learning

Kaiyan Zhang (Wayfair) · Zhiyuan Ma (Huazhong University of Science and Technology) · Ning Ding (Tsinghua University, Tsinghua University) · Ganqu Cui (Shanghai Artificial Intelligence Laboratory) · Lifan Yuan (University of Illinois Urbana-Champaign) · Yuxin Zuo · Li Sheng (Tsinghua University, Tsinghua University) · Shang Qu (University of Science and Technology of China) · Xuekai Zhu · Haozhan Li (Tsinghua University) · yuchen zhang (UESTC) · Xinwei Long (Tsinghua University) · Ermo Hua (Tsinghua University, Tsinghua University) · Biqing Qi (Tsinghua University) · Youbang Sun (Tsinghua University, Tsinghua University) · Bowen Zhou (Tsinghua University)

experimental findingsground-truth labelsmaj@n metricmajority votingpass@1 performanceperformance improvement pre-trained modelsqwen-2.5-math-7breinforcement learningreward estimationself-evolutiontest-time reinforcement learningtest-time scaling unlabeled data

This paper investigates Reinforcement Learning (RL) on data without explicit labels for reasoning tasks in Large Language Models (LLMs). The core challenge of the problem is reward estimation during inference while not having access to ground-truth information. While this setting appears elusive, we find that common practices in Test-Time Scaling (TTS), such as majority voting, yield surprisingly effective rewards suitable for driving RL training. In this work, we introduce Test-Time Reinforcement Learning (TTRL), a novel method for training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs by utilizing the priors in the pre-trained models. Our experiments demonstrate that TTRL consistently improves performance across a variety of tasks and models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by approximately 211% on the AIME 2024 with only unlabeled test data. Furthermore, although TTRL is only supervised by the Maj@N metric, TTRL has demonstrated performance to consistently surpass the upper limit of the initial model, and approach the performance of models trained directly on test data with ground-truth labels. Our experimental findings validate the general effectiveness of TTRL across various tasks and highlight TTRL's potential for broader tasks and domains.