Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training

Bhavya Kailkhura (Lawrence Livermore National Laboratory) · Yoshua Bengio (Mila/U. Montreal) · Brian Bartoldson (Lawrence Livermore National Laboratory) · Johan Obando Ceron (Mila / Université de Montréal) · Minsu Kim (Mila / KAIST) · Siddarth Venkatraman (Mila) · James Diffenderfer (Lawrence Livermore National Laboratory) · Moksh Jain (Mila, Université de Montréal, Google DeepMind) · Tal Ben-Nun (Lawrence Livermore National Laboratory) · Seanie Lee (KAIST)

asynchronous off-policyautomated red-teamingdata generation scalingdr. grpoexperience replayoff-policy dataonline dpoperformance boostspost-trainingpreference-tuningrecency-prioritizing samplingreinforcement learningreward-prioritizing samplingtb objectivetrajectory balance

Reinforcement learning (RL) is a critical component of large language model (LLM) post-training. However, on-policy algorithms used for post-training are not naturally robust to a diversified content of experience replay buffers, which asynchronous off-policy actors can efficiently populate in parallel to training. We propose efficiently learning on such off-policy data via Trajectory Balance with Asynchrony (TBA), an approach to asynchronous RL for LLMs that leverages the principled off-policy TB objective. On math, preference-tuning, and automated red-teaming tasks, we post-train models ranging from Pythia 410M to Qwen 2.5 7B, finding TBA offers speed and performance boosts over strong baselines like Online DPO and Dr. GRPO. Beyond TBA's performance benefits (high accuracy even as asynchrony grows) and speedups ($4\times$ or more), we show its reward- and recency-prioritizing sampling enable further gains as data generation is scaled. Our code is available at https://github.com/bbartoldson/TBA.