LLM at Network Edge: A Layer-wise Efficient Federated Fine-tuning Approach

Nan Cheng (Xi'an University of Electronic Science and Technology) · Haozhao Wang (The Hong Kong Polytechnic University) · Wenchao Xu (Hong Kong University of Science and Technology) · Jiajie Xu (Xidian University) · Jinglong Shen (Xidian University) · Yifan guo (Xidian Universit)

$\mathcal{o}(1/\sqrt{t})$client-side computational overheadcomputational efficiency convergence rate diverse datasets extensive experiments federated learning heterogeneous environmentsimportance-driven layer samplinglayer-wise efficient federated fine-tuningmodel performancestraggler effectsuperior computational efficiencytheoretical analysis

Fine-tuning large language models (LLMs) poses significant computational burdens, especially in federated learning (FL) settings. We introduce Layer-wise Efficient Federated Fine-tuning (LEFF), a novel method designed to enhance the efficiency of FL fine-tuning while preserving model performance and minimizing client-side computational overhead. LEFF strategically selects layers for fine-tuning based on client computational capacity, thereby mitigating the straggler effect prevalent in heterogeneous environments. Furthermore, LEFF incorporates an importance-driven layer sampling mechanism, prioritizing layers with greater influence on model performance. Theoretical analysis demonstrates that LEFF achieves a convergence rate of $\mathcal{O}(1/\sqrt{T})$. Extensive experiments on diverse datasets demonstrate that LEFF attains superior computational efficiency and model performance compared to existing federated fine-tuning methods, particularly under heterogeneous conditions.