Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL

Songjun Tu (Institute of Automation Chinese Academy of Sciences) · Jiahao Lin (University of the Chinese Academy of Sciences) · Qichao Zhang (Institute of automation, Chinese academy of science, Chinese Academy of Sciences) · Xiangyu Tian (University of the Chinese Academy of Sciences) · Linjing Li (Institute of automation, Chinese academy of science, Chinese Academy of Sciences) · Xiangyuan Lan (Department of Computer Science, Hong Kong Baptist University) · Dongbin Zhao (Institute of Automation, Chinese Academy of Sciences)

accuracy–efficiency trade-offsadaptive thinking capabilitiescomputational overheaddistilled modelsexplicit reasoninglarge reasoning modelslatent controllabilitymathematical benchmarksprompting methodsreasoning policiesreinforcement learning frameworkrl-based pruningscalable reasoning paradigmstage-wise reward shapingsuccinct responses

Large reasoning models (LRMs) are proficient at generating explicit, step-by-step reasoning sequences before producing final answers. However, such detailed reasoning can introduce substantial computational overhead and latency, particularly for simple problems. To address this over-thinking problem, we explore how to equip LRMs with adaptive thinking capabilities—enabling them to dynamically decide whether or not to engage in explicit reasoning based on problem complexity. Building on R1-style distilled models, we observe that inserting a simple ellipsis ("...") into the prompt can stochastically trigger either a thinking or no-thinking mode, revealing a latent controllability in the reasoning behavior. Leveraging this property, we propose AutoThink, a multi-stage reinforcement learning (RL) framework that progressively optimizes reasoning policies via stage-wise reward shaping. AutoThink learns to invoke explicit reasoning only when necessary, while defaulting to succinct responses for simpler tasks. Experiments on five mainstream mathematical benchmarks demonstrate that AutoThink achieves favorable accuracy–efficiency trade-offs compared to recent prompting and RL-based pruning methods. It can be seamlessly integrated into any R1-style model, including both distilled and further fine-tuned variants. Notably, AutoThink improves relative accuracy by 6.4\% while reducing token usage by 52\% on DeepSeek-R1-Distill-Qwen-1.5B, establishing a scalable and adaptive reasoning paradigm for LRMs. Project Page: https://github.com/ScienceOne-AI/AutoThink.