TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

Graham Neubig (Carnegie Mellon University) · Mingyang Yang (Carnegie Mellon University) · Frank (Fangzheng) Xu (Microsoft AI) · Yufan Song (Carnegie Mellon University) · Boxuan Li (Microsoft) · Yuxuan Tang (Oracle) · Kritanjali Jain (School of Computer Science, Carnegie Mellon University) · Mengxue Bao (Tiktok) · Zora Wang (Carnegie Mellon University) · Xuhui Zhou (CMU, Carnegie Mellon University) · Zhitong Guo (Meta) · Murong Cao (University of Hong Kong) · Hao Yang Lu (Carnegie Mellon University) · Amaad Martin (School of Computer Science, Carnegie Mellon University) · Zhe Su (Carnegie Mellon University) · Leander Maben (CMU, Carnegie Mellon University) · Raj Mehta (Carnegie Mellon University) · Wayne Chi (Carnegie Mellon University) · Lawrence Jang (Carnegie Mellon University) · Yiqing Xie (Carnegie Mellon University) · Shuyan Zhou (Facebook)
ai agentsautonomous task completionbaseline agentsclosed apidigital workereconomic policy implicationsextensible benchmarkinteraction with environmentslabor market effectsopen-weights modelsperformance evaluationreal-world professional taskssoftware company simulationtask automationtask complexity

We interact with computers on an everyday basis, be it in everyday life or work, and many aspects of work can be done entirely with access to a computer and the Internet. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments. But how performant are AI agents at helping to accelerate or even autonomously perform work-related tasks? The answer to this question has important implications for both industry looking to adopt AI into their workflows, and for economic policy to understand the effects that adoption of AI may have on the labor market. To measure the progress of these LLM agents' performance on performing real-world professional tasks, in this paper, we introduce TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and communicating with other coworkers. We build a self-contained environment with internal web sites and data that mimics a small software company environment, and create a variety of tasks that may be performed by workers in such a company. We test baseline agents powered by both closed API-based and open-weights language models (LMs), and find that with the most competitive agent, 30% of the tasks can be completed autonomously. This paints a nuanced picture on task automation with LM agents -- in a setting simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems. For more information and demos, refer to https://the-agent-company.com.