Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback

Yaodong Yang (AIG) · Rui Pan (Beijing University of Posts and Telecommunications) · Han Zhu (Alibaba Group) · Sirui Han (The Hong Kong University of Science and Technology) · Boyuan Chen (Peking University) · Donghai Hong (Peking University) · Jiaming Ji (Peking University) · Jiayi Zhou (Peking University) · Kaile Wang (Peking University) · Juntao Dai (Peking University) · Yike Guo (Imperial College London) · Xinyu Chen (nanjing university) · Jiahao Li (Microsoft Research Asia) · Chi-Min Chan (Hong Kong University of Science and Technology)

adversarial attacks constrained optimizationdual preference annotationsexperimental results fine-tuningguardrail systemmin-max optimizationmodel helpfulnessmulti-level safety labelsmultimodal large language models multimodal models optimization processpreference signalssafe rlhf-vsafety alignment safety constraints

Multimodal large language models (MLLMs) are essential for building general-purpose AI assistants; however, they pose increasing safety risks. How can we ensure safety alignment of MLLMs to prevent undesired behaviors? Going further, it is critical to explore how to fine-tune MLLMs to preserve capabilities while meeting safety constraints. Fundamentally, this challenge can be formulated as a min-max optimization problem. However, existing datasets have not yet disentangled single preference signals into explicit safety constraints, hindering systematic investigation in this direction. Moreover, it remains an open question whether such constraints can be effectively incorporated into the optimization process for multi-modal models. In this work, we present the first exploration of the Safe RLHF-V -- the first multimodal safety alignment framework. The framework consists of: (I) BeaverTails-V, the first open-source dataset featuring dual preference annotations for helpfulness and safety, supplemented with multi-level safety labels (minor, moderate, severe); (II) Beaver-Guard-V, a multi-level guardrail system to proactively defend against unsafe queries and adversarial attacks. Applying the guard model over five rounds of filtering and regeneration significantly enhances the precursor model’s overall safety by an average of 40.9%. (II) Based on dual preference, we initiate the first exploration of multi-modal safety alignment within a constrained optimization. Experimental results demonstrate that Safe RLHF effectively improves both model helpfulness and safety. Specifically, Safe RLHF-V enhances model safety by 34.2% and helpfulness by 34.3%.