Ask a Strong LLM Judge when Your Reward Model is Uncertain

Tuo Zhao (Georgia Tech) · Qingru Zhang (Microsoft CoreAI) · Zhenghao Xu (Georgia Institute of Technology) · Qin Lu (Amazon) · Liang Qiu (Amazon) · Ilgee Hong (Georgia Institute of Technology) · Changlong Yu (Department of Computer Science and Engineering, The Hong Kong University of Science and Technology) · Wenlin Yao (Amazon) · Yao Liu (Stanford University) · Haoming Jiang (Georgia Institute of Technology) · Lihong Li (Amazon) · Hyokun Yun (Amazon)

advantage estimationdownstream alignmenthuman feedback inference costs online rlhfout-of-distribution inputspairwise preference classificationpolicy gradient methodsrandom judge callingreinforcement learning reward hacking reward model uncertainty quantificationuncertainty-based routing

Reward model (RM) plays a pivotal role in reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs). However, classical RMs trained on human preferences are vulnerable to reward hacking and generalize poorly to out-of-distribution (OOD) inputs. By contrast, strong LLM judges equipped with reasoning capabilities demonstrate superior generalization, even without additional training, but incur significantly higher inference costs, limiting their applicability in online RLHF. In this work, we propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge. Our approach formulates advantage estimation in policy gradient (PG) methods as pairwise preference classification, enabling principled uncertainty quantification to guide routing. Uncertain pairs are forwarded to the LLM judge, while confident ones are evaluated by the RM. Experiments on RM benchmarks demonstrate that our uncertainty-based routing strategy significantly outperforms random judge calling at the same cost, and downstream alignment results showcase its effectiveness in improving online RLHF.