You Only Communicate Once: One-shot Federated Low-Rank Adaptation of MLLM

Rui Yan (Nanjing University of Science and Technology) · Basura Fernando (Nanyang Technological University) · Xiangbo Shu (Nanjing University Science and Technology) · Yazhou Yao (Nanjing University of Science and Technology) · Binqian Xu (Nanjing University of Science and Technology) · Haiyang Mei · Zechen Bai (Meta; NUS) · Jinjin Gong (Nanjing University of Science and Technology) · Guosen Xie (inception institute of artificial intelligence (iiai))

adaptive ensemble methodsclient conflictsclient-specific adaptabilitycommunication efficiencydirectional supervisionempirical study federated learning global consistencyglobal supervisionlocal bias correctionmultimodal large language models multimodal scenarios one-shot federated learningsign-regularized lorasparsely regularized lora

Multimodal Large Language Models (MLLMs) with Federated Learning (FL) can quickly adapt to privacy-sensitive tasks, but face significant challenges such as high communication costs and increased attack risks, due to their reliance on multi-round communication. To address this, One-shot FL (OFL) has emerged, aiming to complete adaptation in a single client-server communication. However, existing adaptive ensemble OFL methods still need more than one round of communication, because correcting heterogeneity-induced local bias relies on aggregated global supervision, meaning they still do not achieve true one-shot communication. In this work, we make the first attempt to achieve true one-shot communication for MLLMs under OFL, by investigating whether implicit (i.e., initial rather than aggregated) global supervision alone can effectively correct local training bias. Our key finding from the empirical study is that imposing directional supervision on local training substantially mitigates client conflicts and local bias. Building on this insight, we propose YOCO, in which directional supervision with sign-regularized LoRA B enforces global consistency, while sparsely regularized LoRA A preserves client-specific adaptability. Experiments demonstrate that YOCO cuts communication to $\sim$0.03\% of multi-round FL while surpassing those methods in several multimodal scenarios and consistently outperforming all one-shot competitors.