MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research

Yue Liu (NUS) · Bryan Hooi (National University of Singapore) · Hui Chen (National University of Singapore) · Miao Xiong (Amazon ) · Yujie Lu (Meta) · Wei Han (Huawei Tech. Investment Co., Limited) · Ailin Deng (National University of Singapore / Meta) · Yufei He (National University of Singapore) · Jiaying Wu (National University of Singapore) · Yibo Li (National University of Singapore (NUS))

ai agentsautomated evaluationcoding agentscoherent ideasend-to-end evaluationexperimentationidea generationllm-based reviewersmachine learning researchmlr-agentmlr-benchmlr-judgeproposal formulationreview rubricsscientific reliabilitytrustworthy scientific discovery

Recent advancements in AI agents have demonstrated their growing potential to drive and support scientific discovery. In this work, we introduce MLR-Bench, a comprehensive benchmark for evaluating AI agents on open-ended machine learning research. MLR-Bench includes three key components: (1) 201 research tasks sourced from NeurIPS, ICLR, and ICML workshops covering diverse ML topics; (2) MLR-Judge, an automated evaluation framework combining LLM-based reviewers with carefully designed review rubrics to assess research quality; and (3) MLR-Agent, a modular agent scaffold capable of completing research tasks through four stages: idea generation, proposal formulation, experimentation, and paper writing. Our framework supports both stepwise assessment across these distinct research stages, and end-to-end evaluation of the final research paper. We then use MLR-Bench to evaluate six frontier LLMs and an advanced coding agent, finding that while LLMs are effective at generating coherent ideas and well-structured papers, current coding agents frequently (e.g., in 80\% of the cases) produce fabricated or invalidated experimental results—posing a major barrier to scientific reliability. We validate MLR-Judge through human evaluation, showing high agreement with expert reviewers, supporting its potential as a scalable tool for research evaluation. We open-source MLR-Bench to help the community benchmark, diagnose, and improve AI research agents toward trustworthy and transparent scientific discovery.