Researchers From Stanford And DeepMind Come Up With The Idea of Using Large Language Models LLMs as a Proxy Reward Function

Abstraction: LLMs as proxy reward functions for RL agent alignment via natural language

Key points:

• Stanford and DeepMind researchers propose using a prompted LLM as a stand-in reward function instead of manually coded rewards or large labeled datasets
• Users specify goals conversationally (e.g., one sentence or a few examples); the LLM scores RL episode trajectories against the stated goal at each step
• Zero-shot prompting increased objective-aligned reward signals by 48% on regular matrix game outcomes and 36% on scrambled ordering compared to baseline
• Tested on Ultimatum Game, DEALORNODEAL negotiation, and MatrixGames with only a few prompts; pilot used 10 human participants
• Key benefit: LLMs encode commonsense priors about human behavior, reducing the need for large labeled datasets; single correct outcome detection is sufficient
• Code available at github.com/minaek/reward_design_with_llms

Connections: Stanford University · Deepmind · Reinforcement Learning · Large Language Models · Reward Modeling

Source: https://www.marktechpost.com/2023/07/20/researchers-from-stanford-and-deepmind-come-up-with-the-idea-of-using-large-language-models-llms-as-a-proxy-reward-function/