The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements

Oisin Mac Aodha (University of Edinburgh) · Jakob Foerster (University of Oxford) · Thomas Foster (University of OxfordFAIR @) · Xian Li (Meta MSL) · Edan Toledo (Meta | UCL) · Karen Hambardzumyan (Meta) · Martin Josifoski (Facebook) · Despoina Magka (FAIR at Meta) · Minqi Jiang (UCL & FAIR) · Alisia Lupidi (University of Cambridge) · Andrei Lupu (University of Oxford, Meta AI) · Roberta Raileanu (Meta) · Tatiana Shavrina (Meta) · Kelvin Niu (Facebook) · Jean-Christophe Gagnon-Audet (Montreal Institute for Learning Algorithms, University of Montreal, University of Montreal) · Michael Shvartsman (Meta FAIR) · Shagun Sodhani (Google DeepMind) · Alexander Miller (Meta) · Abhishek Charnalia (Facebook) · Derek Dunfield (Facebook) · Yoram Bachrach (Google DeepMind) · Bingchen Zhao (University of Edinburgh) · Lucia Cipolina Kun (University of Bristol)
ai agentsalgorithmic advancementsautomated llm speedrunning benchmarkautonomous research agentcode-level changesfrontier reasoninghardware-aware optimizationshint formatsnanogpt speedrunnon-saturated measurepseudocodereimplementationscientific reproductionsota scaffoldstraining script

Rapidly improving large language models (LLMs) have the potential to assist in scientific progress. One critical skill in this endeavor is the ability to faithfully reproduce existing work. To evaluate the capability of AI agents to reproduce complex code in an active research area, we introduce the Automated LLM Speedrunning Benchmark, leveraging the research community's contributions to the $\textit{NanoGPT speedrun}$, a competition to train a GPT-2 model in the shortest time. Each of the 19 speedrun tasks provides the agent with the previous record's training script, optionally paired with one of three hint formats, ranging from pseudocode to paper-like descriptions of the new record's improvements. Records execute quickly by design and speedrun improvements encompass diverse code-level changes, ranging from high-level algorithmic advancements to hardware-aware optimizations. These features make the benchmark both accessible and realistic for the frontier problem of improving LLM training. We find that recent frontier reasoning LLMs combined with SoTA scaffolds struggle to reimplement already-known innovations in our benchmark, even when given detailed hints. Our benchmark thus provides a simple, non-saturated measure of an LLM's ability to automate scientific reproduction, a necessary (but not sufficient) skill for an autonomous research agent.