Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning

Fan Chen (Massachusetts Institute of Technology) · Zeyu Jia (Massachusetts Institute of Technology) · Alexander Rakhlin (MIT) · Tengyang Xie (UW-Madison) · Yurun Yuan (Department of Computer Science, University of Wisconsin - Madison)

bellman residual minimizationchange-of-trajectory-measure analysiscomputational overhead convergence grpokl-regularized policymathematical-reasoning benchmarksmemory overhead off-policy algorithmpolicy-based baselinesppo q-values reinforcement learningtrajectory bellman residual minimizationtrajectory-level bellman objective

Policy-based methods currently dominate reinforcement learning (RL) pipelines for large language model (LLM) reasoning, leaving value-based approaches largely unexplored. We revisit the classical paradigm of Bellman Residual Minimization and introduce Trajectory Bellman Residual Minimization (TBRM), an algorithm that naturally adapts this idea to LLMs, yielding a simple yet effective off-policy algorithm that optimizes a single trajectory-level Bellman objective using the model's own logits as $Q$-values. TBRM removes the need for critics, importance-sampling ratios, or clipping, and can operate with only one rollout per prompt. We prove convergence to the near-optimal KL-regularized policy from arbitrary off-policy data via an improved change-of-trajectory-measure analysis. Experiments on standard mathematical-reasoning benchmarks show that TBRM matches or surpasses policy-based baselines, like PPO and GRPO, with comparable or lower computational and memory overhead. Our results indicate that value-based RL might be a principled and efficient alternative for enhancing reasoning capabilities in LLMs. The codebase for TBRM is publicly available at [https://github.com/rlx-lab/TBRM](https://github.com/rlx-lab/TBRM).