CHPO: Constrained Hybrid-action Policy Optimization for Reinforcement Learning

Sanqing Qu (Tongji University) · Fan Lu (Tongji University) · Ya Wu (CNNC) · Guang Chen (Tongji University) · Li Shen (Sun Yat-Sen University) · Ziqiao Wang (Tongji University) · Junqiao Zhao (Tongji University) · ao zhou (Tongji university) · Jiayi Guan (Tongji University)

algorithm convergence.competitive performanceconstrained hybrid-action policy optimizationconstrained hybrid-action reinforcement learningconstrained hybrid-action rlconstrained parameterized-action markov decision processcost constraintsexperimental taskshybrid action spacesoptimal solutionparameterized action spacereward maximization safety constraints safety-critical applications theoretical analyses

Constrained hybrid-action reinforcement learning (RL) promises to learn a safe policy within a parameterized action space, which is particularly valuable for safety-critical applications involving discrete-continuous hybrid action spaces. However, existing hybrid-action RL algorithms primarily focus on reward maximization, which faces significant challenges for tasks involving both cost constraints and hybrid action spaces. In this work, we propose a novel Constrained Hybrid-action Policy Optimization algorithm (CHPO) to address the problems of constrained hybrid-action RL. Concretely, we rethink the limitations of hybrid-action RL in handling safe tasks with parameterized action spaces and reframe the objective of constrained hybrid-action RL by introducing the concept of Constrained Parameterized-action Markov Decision Process (CPMDP). Subsequently, we present a constrained hybrid-action policy optimization algorithm to confront the constrained hybrid-action problems and conduct theoretical analyses demonstrating that the CHPO converges to the optimal solution while satisfying safety constraints. Finally, extensive experiments demonstrate that the CHPO achieves competitive performance across multiple experimental tasks.