Establishing Best Practices in Building Rigorous Agentic Benchmarks

Sayash Kapoor (Princeton University) · Shayne Longpre (Massachusetts Institute of Technology) · Percy Liang (Stanford University) · Ion Stoica (UC Berkeley) · Rahul Gupta (Amazon.com) · Jacob Steinhardt (UC Berkeley, Transluce AI) · Yuxuan Zhu (Rensselaer Polytechnic Institute) · Tengjun Jin (University of Illinois at Urbana-Champaign) · Yada Pruksachatkun (Salesforce Research) · Andy Zhang (Stanford / Berkeley) · Shu Liu (University of California, Berkeley) · Sasha Cui (Yale University) · Kevin Meng (MIT) · Rebecca Weiss (MLCommons Association ) · Fazl Barez (University of Oxford) · Jwala Dhamala (Amazon Alexa AI) · Jacob Merizian (Virginia Polytechnic Institute and State University) · Mario Giulianelli (University College London, University of London) · Harry Coppock (Imperial College London, Number 10 Downing Street) · Cozmin Ududec (UK AI Security Institute) · Antony Kellermann (University of Illinois at Urbana-Champaign) · Jasjeet Sekhon (Yale University) · Sarah Schwettmann (Transluce) · Arvind Narayanan (Princeton University) · Matei A Zaharia (UC Berkeley) · Daniel Kang (UIUC)
$\tau$-benchagentic benchmark checklistagentic benchmarksbenchmark-building experiencebest practicescomplex tasksevaluation designperformance estimationperformance overestimationquantitative trackingreward designsrigorous evaluationswe-bench-verifiedtask outcomestest cases

Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to evaluate agents on complex, real-world tasks. These benchmarks typically measure agent capabilities by evaluating task outcomes via specific reward designs. However, we show that many agentic benchmarks have issues in task setup or reward design. For example, SWE-bench-Verified uses insufficient test cases, while $\tau$-bench counts empty responses as successes. Such issues can lead to under- or overestimation of agents’ performance by up to 100% in relative terms. To make agentic evaluation rigorous, we introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues. When applied to CVE-Bench, a benchmark with a particularly complex evaluation design, ABC reduces performance overestimation by 33%.