AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?

Haoyu Zhao (Princeton University) · Ni Zhan (Princeton University) · Arman Zharmagambetov (FAIR @ Meta) · Kilian Lieret (Princeton University) · Ofir Press · Matthias Bethge (University of Tübingen) · Brandon Amos (Startup) · Hanlin Zhang (Harvard University) · Ori Press (University of Tuebingen) · Yikai Wu (Princeton University) · Samuel Ainsworth (Waymo) · Dominik Krupke (Technische Universität Carolo-Wilhelmina zu Braunschweig) · Patrick Kidger (Google X) · Touqir Sajed (LG Electronics Canada Inc.) · Bartolomeo Stellato (Princeton University) · Jisun Park (Princeton University) · Nathanael Bosch (University of Tuebingen) · Eli Meril (Tel Aviv University) · Albert Steppi (Quansight Labs) · Fangzhao Zhang (Stanford University) · David Pérez-Piñeiro (Gridmatic) · Alberto Mercurio (EPFL - EPF Lausanne) · Talor Abramovich (School of Computer Science, Tel Aviv University) · Shirley Huang (Harvard University)
algorithm designalgorithmic innovationscode synthesiscomputational problemscreative problem solvingdomain expertsfrontier modelslanguage modelslm agentsopen-ended benchmarkperformance evaluationreference implementationsspeedupsurface-level optimizationsvalidation framework

Despite progress in language model (LM) capabilities, evaluations have thus far focused on models' performance on tasks that humans have previously solved, including in programming (SWE-Bench) and mathematics (FrontierMath). We therefore propose testing models' ability to design and implement algorithms in an open-ended benchmark: We task LMs with writing code that efficiently solves computationally challenging problems in computer science, physics, and mathematics. Our AlgoTune benchmark consists of 120 tasks collected from domain experts and a framework for validating and timing LM-synthesized solution code, which is compared to reference implementations from popular open-source packages.In addition, we develop a baseline LM agent, AlgoTuner, and evaluate its performance across a suite of frontier models.AlgoTuner achieves an average 1.58x speedup against reference solvers, including methods from packages such as SciPy, scikit-learn and CVXPY.However, we find that current models fail to discover algorithmic innovations, instead preferring surface-level optimizations. We hope that AlgoTune catalyzes the development of LM agents exhibiting creative problem solving beyond state-of-the-art human performance.