MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?

Lu Wang (University of Michigan) · Yunxiang Zhang (University of Michigan) · Muhammad Khalifa (University of Michigan - Ann Arbor) · Shitanshu Bhushan (University of Michigan - Ann Arbor) · Grant Murphy (University of Michigan - Ann Arbor) · Lajanugen Logeswaran (LG AI Research) · Jaekyeom Kim (LG AI Research) · Moontae Lee (LG Corporation) · Honglak Lee (LG AI Research / U. Michigan)

agentic pipelineai research capabilitiescompetition tasksdynamic benchmarkinnovation misalignmentlanguage agentsleaderboard.llm agents llm-as-a-judgemachine learning competitionsmlrc-benchnovel methodologiesobjective metrics performance evaluationresearch methodsrigorous protocol

We introduce **MLRC-Bench**, a benchmark designed to quantify how effectively language agents can tackle challenging **M**achine **L**earning (ML) **R**esearch **C**ompetitions, with a focus on open research problems that demand novel methodologies.Unlike prior work, e.g., AI Scientist, which evaluates the end-to-end agentic pipeline by using LLM-as-a-judge, MLRC-Bench measures the key steps of proposing and implementing novel research methods and evaluates them with rigorous protocol and objective metrics.Our curated suite of 7 competition tasks reveals significant challenges for LLM agents. Even the best-performing tested agent (gemini-exp-1206 under MLAB) closes only 9.3% of the gap between baseline and top human participant scores.Furthermore, our analysis reveals a misalignment between the *LLM-judged* innovation and their *actual* performance on cutting-edge ML research problems. MLRC-Bench is a dynamic benchmark, which is designed to continually grow with new ML competitions to encourage rigorous and objective evaluations of AI’s research capabilities. Our leaderboard and code are publicly available at https://huggingface.co/spaces/launch/MLRC_Bench.