Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

Bo Wang (Sensetime) · Qinyuan Cheng (Fudan University) · Runyu Peng (Fudan University) · Rong Bao (Fudan University) · Peiji Li (Fudan University) · Qipeng Guo (AWS) · Linyang Li (Fudan University) · Zhiyuan Zeng (University of Washington) · Yunhua Zhou (Shanghai Artificial Intelligence Laboratory) · Xipeng Qiu (Fudan University)

direct preference optimization distribution matchingf-divergence functionsimplicit reward learninginstruction following taskskl divergence large language modellearning rate reductionmathematical derivationsmodel performanceoptimal policy-reward subspacepost-training processespre-trained language models preference learningq-functionssupervised fine-tuning

Post-training processes are essential phases in grounding pre-trained language models to real-world tasks, with learning from demonstrations or preference signals playing a crucial role in this adaptation. We present a unified theoretical framework bridging Supervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training. Through rigorous mathematical derivation, we demonstrate that both SFT and preference learning methods like Direct Preference Optimization (DPO) operate within the same optimal policy-reward subspace, with SFT representing a special case of implicit reward learning. Our analysis reveals a critical limitation in conventional SFT: the KL divergence term in distribution matching becomes constant with respect to the policy during optimization, failing to constrain model updates. To address this, we propose a simple yet effective learning rate reduction approach that yields significant performance improvements (up to \textbf{25\%} relative gain and \textbf{6\%} absolute win rate increase in instruction following tasks. Additionally, we derive alternative SFT objectives from various f-divergence functions that preserve the KL term during optimization, further enhancing post-DPO model performance. Finally, we extend the theoretical relationship between LLM logits and Q-functions from preference learning to the SFT context, providing mathematical derivations and experimental validation.