Towards Building Model/Prompt-Transferable Attackers against Large Vision-Language Models

Pan Zhou (Singapore Management University) · Lichao Sun (Lehigh University) · Wei Hu (University of Michigan) · Xiaowen Cai (Huazhong University of Science and Technology) · Daizong Liu (Wuhan University) · Xiaoye Qu (Shanghai Artificial Intelligence Laboratory) · Jianfeng Dong (Zhejiang Gongshang University) · Xiang Fang (Huazhong University of Science and Technology) · Keke Tang (Guangzhou University)

adversarial examples adversarial learningagnostic perturbationsblack-box modelsdisentangled patternsinformation gain information theory large vision-language modelsmisleading perceptionmultimodal capabilities mutual informationsecurity concernstask-aware promptstransfer-attack performancetransferable attacks

Although Large Vision-Language Models (LVLMs) exhibit impressive multimodal capabilities, their vulnerability to adversarial examples has raised serious security concerns. Existing LVLM attackers simply optimize adversarial images that easily overfit a certain model/prompt, making them ineffective once they are transferred to attack a different model/prompt. Motivated by this research gap, this paper aims to develop a more powerful attack that is transferable to black-box LVLM models of different structures and task-aware prompts of different semantics. Specifically, we introduce a new perspective of information theory to investigate LVLMs' transferable characteristics by exploring the relative dependence between outputs of the LVLM model and input adversarial samples. Our empirical observations suggest that enlarging/decreasing the mutual information between outputs and the disentangled adversarial/benign patterns of input images helps to generate more agnostic perturbations for misleading LVLMs' perception with better transferability. In particular, we formulate the complicated calculation of information gain as an estimation problem and incorporate such informative constraints into the adversarial learning process. Extensive experiments on various LVLM models/prompts demonstrate our significant transfer-attack performance.