ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering

Yuki Imajuku (Sakana AI) · Takuya Akiba (Sakana AI) · Kohki Horie (The University of Tokyo/Sakana AI) · Yoichi Iwata (AtCoder Inc.) · Kensho Aoki (AtCoder Inc.) · Naohiro Takahashi (AtCoder Inc.)

ale-benchalgorithm engineeringcomputational hardnessconsistency across problemscrew schedulingfactory production planningfrontier llmsinteractive agent architecturesiterative solution refinementlong-horizon problem-solving capabilitiesoptimization problemspackage-delivery routingpower-grid balancingscore-based algorithmic programmingtest-run feedback

How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing?We introduce $\textit{ALE-Bench}$, a new benchmark for evaluating AI systems on score-based algorithmic programming contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench presents optimization problems that are computationally hard and admit no known exact solution.Unlike short-duration, pass/fail coding benchmarks, ALE-Bench encourages iterative solution refinement over long time horizons.Our software framework supports interactive agent architectures that leverage test-run feedback and visualizations. Our evaluation of frontier LLMs revealed that while they demonstrate high performance on specific problems, a notable gap remains compared to humans in terms of consistency across problems and long-horizon problem-solving capabilities. This highlights the need for this benchmark to foster future AI advancements.