Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm

Jiaqi Zhang (Chongqing University) · Libo Zhang (University of Auckland) · Yang Chen (Shanghai AI Lab) · Menglin Zou (University of Auckland) · Yitan Zhang (University of Auckland) · Junyi Yang (University of Auckland) · Gaël Gendron (LawZero, Mila - Quebec AI Institute) · Jiamou Liu (University of Auckland) · Michael Witbrock (The University of Auckland)

adversarial formulationenergy-based formulationsformal guarantees inverse reinforcement learningminorization-maximization processmonotonic improvementnon-adversarial approachespolicy imitationpolicy optimizationproximal inverse reward optimizationreward functionreward recoverystability guaranteestrust region policy optimizationtrust region reward optimizationunstable training

Inverse Reinforcement Learning (IRL) learns a reward function to explain expert demonstrations. Modern IRL methods often use the adversarial (minimax) formulation that alternates between reward and policy optimization, which often lead to {\em unstable} training. Recent non-adversarial IRL approaches improve stability by jointly learning reward and policy via energy-based formulations but lack formal guarantees. This work bridges this gap. We first present a *unified* view showing canonical non-adversarial methods explicitly or implicitly maximize the likelihood of expert behavior, which is equivalent to minimizing the expected return gap. This insight leads to our main contribution: *Trust Region Reward Optimization* (TRRO), a framework that guarantees *monotonic* improvement in this likelihood via a Minorization-Maximization process. We instantiate TRRO into *Proximal Inverse Reward Optimization* (PIRO), a practical and stable IRL algorithm. Theoretically, TRRO provides the IRL counterpart to the stability guarantees of Trust Region Policy Optimization (TRPO) in forward RL. Empirically, PIRO matches or surpasses state-of-the-art baselines in reward recovery, policy imitation with high sample efficiency on MuJoCo and Gym-Robotics benchmarks and a real-world animal behavior modeling task.