AI Debate Aids Assessment of Controversial Claims

Hamid Palangi (Google) · Nanyun Peng (University of California, Los Angeles) · Yejin Choi (UW => Stanford / NVIDIA) · Salman Rahman (UCLA) · Sheriff Issaka (University of California, Los Angeles) · Ashima Suvarna (University of California, Los Angeles) · Genglin Liu (University of Illinois Urbana-Champaign) · James Shiffer (University of California, Los Angeles) · Jaeyoung Lee (Seoul National University) · Md Rizwan Parvez (Qatar Computing Research Institute) · Shi Feng (George Washington University) · Julian Michael (New York University) · Liwei Jiang (University of Washington) · Saadia Gabriel (MIT/NYU/UCLA)

ai advisorsai debatebiased judgesclimate change confidence calibrationconsultancy protocolcontroversial claimscovid-19evaluators factual accuracyfrontier ai modelshuman-like personasjudgment accuracymisinformation amplificationscalable oversightsocial divides

As AI grows more powerful, it will increasingly shape how we understand the world. But with this influence comes the risk of amplifying misinformation and deepening social divides—especially on consequential topics where factual accuracy directly impacts well-being. Scalable Oversight aims to ensure AI systems remain truthful even when their capabilities exceed those of their evaluators. Yet when humans serve as evaluators, their own beliefs and biases can impair judgment. We study whether AI debate can guide biased judges toward the truth by having two AI systems debate opposing sides of controversial factuality claims on COVID-19 and climate change where people hold strong prior beliefs. We conduct two studies. Study I recruits human judges with either mainstream or skeptical beliefs who evaluate claims through two protocols: debate (interaction with two AI advisors arguing opposing sides) or consultancy (interaction with a single AI advisor). Study II uses AI judges with and without human-like personas to evaluate the same protocols. In Study I, debate consistently improves human judgment accuracy and confidence calibration, outperforming consultancy by 4-10\% across COVID-19 and climate change claims. The improvement is most significant for judges with mainstream beliefs (up to +15.2\% accuracy on COVID-19 claims), though debate also helps skeptical judges who initially misjudge claims move toward accurate views (+4.7\% accuracy). In Study II, AI judges with human-like personas achieve even higher accuracy (78.5\%) than human judges (70.1\%) and default AI judges without personas (69.8\%), suggesting their potential for supervising frontier AI models. These findings highlight AI debate as a promising path toward scalable, bias-resilient oversight in contested domains.