State Entropy Regularization for Robust Reinforcement Learning

Shie Mannor (Technion) · Yonatan Ashlag (Technion - Israel Institute of Technology, Technion) · Uri Koren (Technion, Google Research) · Mirco Mutti (Technion) · Esther Derman (Technion - Israel Institute of Technology) · Pierre-Luc Bacon (Mila)

formal guarantees performance characterizationpolicy entropy regularizationpolicy evaluation reinforcement learningreward uncertaintyrobust rl methodsrobustness rollouts sample complexityspatially correlated perturbationsstate entropy regularizationstructured perturbations transfer learningtransition uncertainty

State entropy regularization has empirically shown better exploration and sample complexity in reinforcement learning (RL). However, its theoretical guarantees have not been studied. In this paper, we show that state entropy regularization improves robustness to structured and spatially correlated perturbations. These types of variation are common in transfer learning but often overlooked by standard robust RL methods, which typically focus on small, uncorrelated changes. We provide a comprehensive characterization of these robustness properties, including formal guarantees under reward and transition uncertainty, as well as settings where the method performs poorly. Much of our analysis contrasts state entropy with the widely used policy entropy regularization, highlighting their different benefits. Finally, from a practical standpoint, we illustrate that compared with policy entropy, the robustness advantages of state entropy are more sensitive to the number of rollouts used for policy evaluation.