ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning

Jun Wang (iWudao Tech) · Linyi Yang (Southern University of Science and Technology) · Weinan Zhang (Shanghai Jiao Tong University) · Shuyue Hu (Shanghai AI Lab) · Ying Wen (Shanghai Jiao Tong University) · Ziyu Wan (Shanghai Jiao Tong University; Bytedance Seed) · Yunxiang Li (University of British Columbia) · Xiaoyu Wen (Shanghai Jiaotong University) · Yan Song (University College London, University of London) · Hanjing Wang (Shanghai Jiao Tong University) · Mark Schmidt (University of British Columbia)

ablation studies collaboration empirical results generalizationhierarchical agentsiterative reinforcement learningllm-as-a-judge mathematical benchmarksmeta-thinkingmulti-agent reinforcement learning multi-turn interaction reasoningreinforced meta-thinking agentsrobustnessstrategic oversight

Recent research on Reasoning of Large Language Models (LLMs) has sought to further enhance their performance by integrating meta-thinking—enabling models to monitor, evaluate, and control their reasoning processes for more adaptive and effective problem-solving. However, current single-agent work lacks a specialized design for acquiring meta-thinking, resulting in low efficacy. To address this challenge, we introduce Reinforced Meta-thinking Agents (ReMA), a novel framework that leverages Multi-Agent Reinforcement Learning (MARL) to elicit meta-thinking behaviors, encouraging LLMs to think about thinking. ReMA decouples the reasoning process into two hierarchical agents: a high-level meta-thinking agent responsible for generating strategic oversight and plans, and a low-level reasoning agent for detailed executions. Through iterative reinforcement learning with aligned objectives, these agents explore and learn collaboration, leading to improved generalization and robustness. Empirical results from single-turn experiments demonstrate that ReMA outperforms single-agent RL baselines on complex reasoning tasks, including competitive-level mathematical benchmarks and LLM-as-a-Judge benchmarks. Additionally, we further extend ReMA to multi-turn interaction settings, leveraging turn-level ratio and parameter sharing to improve efficiency. Comprehensive ablation studies further illustrate the evolving dynamics of each distinct agent, providing valuable insights into how the meta-thinking reasoning process enhances the reasoning capabilities of LLMs.