Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment

Tianyu Pang (Sea AI Lab) · Chao Du (Sea AI Lab) · Bo Li (Nanjing University) · Xinfeng Li (Nanyang Technological University) · Yang Liu (CUHK) · Yiming Li (Zhejiang University) · Xiaojun Jia (Nanyang Technological University) · Sensen Gao (Mohamed bin Zayed University of Artificial Intelligence) · Simeng Qin (Northeast University at Qinhuangdao Campus) · Yihao Huang (Nanyang Technological University)

adversarial example generationclosed-source modelsclustering techniquescosine similaritydynamic ensemble modelfeature optimal alignmentfoa-attackglobal feature alignmentlocal feature alignmentlocal informationlocal representationsmultimodal large language modelsoptimal transport problempatch tokenstargeted attackstransferable adversarial examples

Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples. While existing methods typically achieve targeted attacks by aligning global features—such as CLIP’s [CLS] token—between adversarial and target samples, they often overlook the rich local information encoded in patch tokens. This leads to suboptimal alignment and limited transferability, particularly for closed-source models. To address this limitation, we propose a targeted transferable adversarial attack method based on feature optimal alignment, called FOA-Attack, to improve adversarial transfer capability. Specifically, at the global level, we introduce a global feature loss based on cosine similarity to align the coarse-grained features of adversarial samples with those of target samples. At the local level, given the rich local representations within Transformers, we leverage clustering techniques to extract compact local patterns to alleviate redundant local features. We then formulate local feature alignment between adversarial and target samples as an optimal transport (OT) problem and propose a local clustering optimal transport loss to refine fine-grained feature alignment. Additionally, we propose a dynamic ensemble model weighting strategy to adaptively balance the influence of multiple models during adversarial example generation, thereby further improving transferability. Extensive experiments across various models demonstrate the superiority of the proposed method, outperforming state-of-the-art methods, especially in transferring to closed-source MLLMs.