Generalizing Experience for Language Agents with Hierarchical MetaFlows

Hao Wang (City University of Hong Kong) · Xinyu Zhang (Harbin Institute Technology, Harbin Institute of Technology) · Yankai Lin (Renmin University of China) · Shengda Fan (Renmin University of China) · Xin Cong (Tsinghua University, Tsinghua University) · Zhong Zhang (Tsinghua University) · Yuepeng Fu (Tsinghua University, Tsinghua University) · Yesai Wu (Tsinghua University) · Enrui Hu (Huawei Technologies Ltd.)

agent tasksappworldexecution costsexecution workflowexperience reusehierarchical experience treehierarchical metaflow mergingmetaflowmetaflowgenperformance improvementreinforcement learning pipelineretrievalsubtaskworkbench

Recent efforts to employ large language models (LLMs) as agents have demonstrated promising results in a wide range of multi-step agent tasks. However, existing agents lack an effective experience reuse approach to leverage historical completed tasks. In this paper, we propose a novel experience reuse framework MetaFlowLLM, which constructs a hierarchical experience tree from historically completed tasks. Each node in this experience tree is presented as a MetaFlow which contains static execution workflow and subtask required by agents to complete dynamically. Then, we propose a Hierarchical MetaFlow Merging algorithm to construct the hierarchical experience tree. When accomplishing a new task, MetaFlowLLM can first retrieve the most relevant MetaFlow node from the experience tree and then execute it accordingly. To effectively generate valid MetaFlows from historical data, we further propose a reinforcement learning pipeline to train the MetaFlowGen. Extensive experimental results on AppWorld and WorkBench demonstrate that integrating with MetaFlowLLM, existing agents (e.g., ReAct, Reflexion) can gain substantial performance improvement with reducing execution costs. Notably, MetaFlowLLM achieves an average success rate improvement of 32.3% on AppWorld and 6.2% on WorkBench, respectively.