InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction

Zijian Zhang (Harbin Institute of Technology) · Caiwen Ding (University of Minnesota - Twin Cities) · Yan Yan (Xiamen University) · Bin Lei (University of Minnesota - Twin Cities) · Weitai Kang (University of Illinois Chicago) · Winson Chen (University of Minnesota - Twin Cities) · Xi Xie (University of Connecticut) · Shan Zuo (University of Connecticut) · Mimi Xie (University of Texas at San Antonio) · Ali Payani (Cisco) · Mingyi Hong (University of Minnesota)

accuracy gainclaude-computer-usecollaborative problem-solvingdecoupled tasksevaluation scriptsgaiageneralist agentinfantagent-nextmodular architecturemultimodal interactionopen-sourceosworldreal-world benchmarks swe-benchtool-based agentsvision agents

This paper introduces \textsc{InfantAgent-Next}, a generalist agent capable of interacting with computers in a multimodal manner, encompassing text, images, audio, and video. Unlike existing approaches that either build intricate workflows around a single large model or only provide workflow modularity, our agent integrates tool-based and pure vision agents within a highly modular architecture, enabling different models to collaboratively solve decoupled tasks in a step-by-step manner. Our generality is demonstrated by our ability to evaluate not only pure vision-based real-world benchmarks (i.e., OSWorld), but also more general or tool-intensive benchmarks (e.g., GAIA and SWE-Bench). Specifically, we achieve a $\mathbf{7.27\\%}$ accuracy gain over Claude-Computer-Use on OSWorld. Codes and evaluation scripts are included in the supplementary material and will be released as open-source.