Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization

Seunghyun Yoon (Adobe Research) · Subhojyoti Mukherjee (Adobe Systems) · Viet Lai (Adobe Systems) · Raghavendra Addanki (Adobe Research) · Ryan Rossi (Purdue University) · Trung Bui (Adobe Research) · Anup B. Rao (Adobe Inc.) · Jayakumar Subramanian (Adobe Systems) · Branislav Kveton (Adobe Research)

agent reasoningclarifying questionsdirect preference optimization empirical comparisonfixed-length responseshyper-parameterslanguage qualityoffline reinforcement learningoptimized rewardspractical approachquestion-answering policiesreward-weighted fine-tuningstate-of-the-art methods supervised fine-tuningtrajectory dataset

Offline reinforcement learning (RL) is a variant of RL where the policy is learned from a previously collected dataset of trajectories and rewards. In our work, we propose a practical approach to offline RL with large language models (LLMs). We recast the problem as reward-weighted fine-tuning, which can be solved using similar techniques to supervised fine-tuning (SFT). To showcase the value of our approach, we apply it to learning short-horizon question-answering policies of a fixed length, where the agent reasons about potential answers or asks clarifying questions. Our work stands in a stark contrast to state-of-the-art methods in this domain, based on SFT and direct preference optimization, which have additional hyper-parameters and do not directly optimize for rewards. We compare to them empirically, and report major gains in both optimized rewards and language quality.