Faithful Dynamic Imitation Learning from Human Intervention with Dynamic Regret Minimization

Bo Ling (Southeast University) · Zhengyu Gan (Southeast University) · Wanyuan Wang (Southeast University) · Guanyu Gao (Nanjing University of Science and Technology) · Weiwei Wu (Southeast University) · Yan Lyu (Southeast University)

agent-generated databehavior distributiondynamic regret minimizationfaithful dynamic imitation learninghuman-in-the-loop imitation learningmixed data trainingnon-convex optimizationpolicy trajectoriesproxy rewardreal-time interventionsafety performance state-of-the-art performance task successtrajectory distributionsunbiased imitation objective

Human-in-the-loop (HIL) imitation learning enables agents to learn complex behaviors safely through real-time human intervention. However, existing methods struggle to efficiently leverage agent-generated data due to dynamically evolving trajectory distributions and imperfections caused by human intervention delays, often failing to faithfully imitate the human expert policy. In this work, we propose Faithful Dynamic Imitation Learning (FaithDaIL) to address these challenges. We formulate HIL imitation learning as an online non-convex problem and employ dynamic regret minimization to adapt to the shifting data distribution and track high-quality policy trajectories. To ensure faithful imitation of the human expert despite training on mixed agent and human data, we introduce an unbiased imitation objective and achieve it by weighting the behavior distribution relative to the human expert's as a proxy reward. Extensive experiments on MetaDrive and CARLA driving benchmarks demonstrate that FaithDaIL achieves state-of-the-art performance in safety and task success with significantly reduced human intervention data compared to prior HIL baselines.