Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis

Caiming Xiong (Salesforce Research) · Tianbao Xie (the University of Hong Kong, University of Hong Kong) · Jiaqi Deng (The University of Hong Kong) · Xiaochuan Li (Tsinghua University) · Junlin Yang (Tsinghua University) · Haoyuan Wu (University of Hong Kong) · Jixuan Chen (University of California, San Diego) · Wenjing Hu (nanjing university) · Xinyuan Wang (University of Hong Kong) · Yuhui Xu (Salesforce) · Zekun Wang (Harbin Institute of Technology) · Yiheng Xu (University of Hong Kong) · Junli Wang (Alibaba Group) · Doyen Sahoo (Salesforce) · Tao Yu (University of Hong Kong)
ablation studiesagentic capabilitiescompositional generalizationcomputer use grounding datasetelement recognitionfine-grained manipulationgui groundinglayout understandingmulti-perspective decouplingmulti-scale modelsnatural language instructionsosworld-gprecise manipulationsoftware commonsensestate-of-the-art performancetext matching

Graphical user interface (GUI) grounding, the ability to map natural language instructions to specific actions on graphical user interfaces, remains a critical bottleneck in computer use agent development. Current benchmarks oversimplify grounding tasks as short referring expressions, failing to capture the complexity of real-world interactions that require software commonsense, layout understanding, and fine-grained manipulation capabilities. To address these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types including text matching, element recognition, layout understanding, and precise manipulation. Additionally, we synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples through multi-perspective decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its effectiveness by outperforming existing approaches on ScreenSpot-v2, ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved grounding with Jedi directly enhances agentic capabilities of general foundation models on complex computer tasks with state-of-the-art performance, improving from 23% to 51% on OSWorld. Through detailed ablation studies, we identify key factors contributing to grounding performance and verify that combining specialized data for different interface elements enables compositional generalization to novel interfaces. All benchmark, data, checkpoints, and code are open-sourced and available at https://osworld-grounding.github.io.