COLA: Towards Efficient Multi-Objective Reinforcement Learning with Conflict Objective Regularization in Latent Space

Zibin Dong (Tianjin University) · Jianye Hao (Tianjin University) · Pengyi Li (Tianjin University) · Hongyao Tang (Montreal Institute for Learning Algorithms, University of Montreal, Université de Montréal) · YAN ZHENG (Tianjin University) · Yifu Yuan (Tianjin University)

continual policy adjustmentscontrol problemsdiverse preferencesexperimental results knowledge sharing latent representationmulti-objective continuous controlmulti-objective reinforcement learningoptimization conflictsregularization term sample complexity shared latent space state-of-the-art baselinesvalue estimationvalue function approximation

Many real-world control problems require continual policy adjustments to balance multiple objectives, which requires the acquisition of high-quality policies to cover diverse preferences. Multi-Objective Reinforcement Learning (MORL) provides a general framework to solve such problems. However, current MORL methods suffer from high sample complexity, primarily due to the neglect of efficient knowledge sharing and conflicts in optimization with different preferences. To this end, this paper introduces a novel framework, Conflict Objective Regularization in Latent Space (**COLA**). To enable efficient knowledge sharing, COLA establishes a shared latent representation space for common knowledge, which can avoid redundant learning under different preferences. Besides, COLA introduces a regularization term for the value function to mitigate the negative effects of conflicting preferences on the value function approximation, thereby improving the accuracy of value estimation. The experimental results across various multi-objective continuous control tasks demonstrate the significant superiority of COLA over the state-of-the-art MORL baselines. Code is available at https://github.com/yeshenpy/COLA.