Enhancing GUI Agent with Uncertainty-Aware Self-Trained Evaluator

Miao Zhang (Aalborg University) · Liqiang Nie (Harbin Institute of Technology (Shenzhen)) · Weili Guan (Harbin Institute of Technology (Shenzhen)) · Gongwei Chen (Harbin Institute of Technology, Shenzhen) · Lirong Jie (Harbin Institute of Technology, Shenzhen) · Lexiao Zou (Harbin Institute of Technology(Shenzhen))

evaluator learninggraphical user interface agentsgroup policy optimizationin-domain and out-of-domain datasetsjudgment capabilitieslightweight mllmsmultimodal large language modelsnavigation trajectoriespositive and negative samplesprompting-based evaluatorsreasoning capabilities self-improvementtrajectory quality assessmentuncertainty-aware reinforced self-traininguncertainty-aware sampling strategy

Benefiting from the availability of extensive navigation trajectories, both manually and automatically annotated, current graphical user interface (GUI) agents have achieved remarkable advancements in performance. However, these annotated datasets often contain substantial noise, which impedes effective agent training and underscores the necessity for rigorous trajectory quality assessment. In contrast to existing prompting-based evaluators that rely on proprietary multimodal large language models (MLLMs), we propose an Uncertainty-aware Reinforced Self-Training (URST) framework to train lightweight MLLMs for efficient and reliable trajectory evaluation. URST iteratively fine-tunes MLLMs using their own generated thoughts and judgments to enable self-improvement, while its uncertainty-aware sampling strategy ensures the selection of the most informative training examples. To further enhance reasoning and judgment capabilities, we propose a simplified group policy optimization approach that effectively leverages diverse positive and negative samples for evaluator learning. Our evaluator demonstrates superior judgment performance across both in-domain and out-of-domain datasets. When used to filter navigation datasets, it consistently leads to performance improvements in training GUI agents.