Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning

Tim Althoff (University of Washington) · Sergey Levine (UC Berkeley) · Marwa Abdulhai (University of California, Berkeley) · Natasha Jaques (University of Washington, Google DeepMind) · Ryan Cheng (UC Berkeley / Berkeley AI Research (BAIR)) · Donovan Clay (University of Washington)

automatic metricscoherent dialoguedialogue evaluationfine-tuning human annotationsline-to-line consistencymulti-turn reinforcement learningpersona consistencypersona driftprompt-to-line consistencyq&a consistencyscalable trainingtrustworthy simulationsuser roles

Large Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play. While these simulations enable scalable training and evaluation of AI agents, off-the-shelf LLMs often drift from their assigned personas, contradict earlier statements, or abandon role-appropriate behavior. We introduce a unified framework for evaluating and improving persona consistency in LLM-generated dialogue. We define three automatic metrics—prompt-to-line consistency, line-to-line consistency, and Q\&A consistency—that capture different types of persona drift and validate each against human annotations. Using these metrics as reward signals, we apply multi-turn reinforcement learning to fine-tune LLMs for three user roles: a patient, a student, and a social chat partner. Our method reduces inconsistency by over 55%, resulting in more coherent, faithful, and trustworthy simulated users.