Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervision

Yu Cheng (The Chinese University of Hong Kong) · Zhi Wang (SIGS, Tsinghua University) · Chunlin Chen (Nanjing University) · Wenhao Wu (nanjing university) · Zican Hu (Nanjing University) · Daoyi Dong (University of Technology Sydney) · Shilin Zhang (nanjing university) · Xinyi Xie (Nanjing University) · Jianxiang Tang (nanjing university) · Zhenhong Sun (Australian National University)

clipcontrastive language-decision pre-trainingdecision tasksdynamics-aware embeddinghigh-capacity zero-shot generalizationmeta-world benchmarksmujoco benchmarksoffline meta-rlraw textsupervision signalstask beliefstext-conditioned generalist policytext-to-decision agentwarmup explorationszero-shot text-to-decision generation

Offline meta-RL usually tackles generalization by inferring task beliefs from high-quality samples or warmup explorations. The restricted form limits their generality and usability since these supervision signals are expensive and even infeasible to acquire in advance for unseen tasks. Learning directly from the raw text about decision tasks is a promising alternative to leverage a much broader source of supervision. In the paper, we propose **T**ext-to-**D**ecision **A**gent (**T2DA**), a simple and scalable framework that supervises offline meta-RL with natural language. We first introduce a generalized world model to encode multi-task decision data into a dynamics-aware embedding space. Then, inspired by CLIP, we predict which textual description goes with which decision embedding, effectively bridging their semantic gap via contrastive language-decision pre-training and aligning the text embeddings to comprehend the environment dynamics. After training the text-conditioned generalist policy, the agent can directly realize zero-shot text-to-decision generation in response to language instructions. Comprehensive experiments on MuJoCo and Meta-World benchmarks show that T2DA facilitates high-capacity zero-shot generalization and outperforms various types of baselines. Our code is available at [https://github.com/NJU-RL/T2DA](https://github.com/NJU-RL/T2DA).