Federated Continual Learning via Orchestrating Multi-Scale Expertise

Yang Liu (CUHK) · Jian Zhang (Xiamen University) · Xiaoyang Yi (Nankai University) · Binhan Yang (Nankai University)

catastrophic forgettingexpert communicationfederated continual learning fine-tuning knowledge fusion lightweight adaptersmulti-expert trainingmulti-modal initializationmulti-scale feature learningmulti-teacher dynamic self-distillationplasticity pre-trained models semantic features stabilitytask class prototypes

Federated continual learning (FCL) aims to maintain the model's performance on old tasks (i.e., stability) while enhancing its ability to acquire knowledge from current tasks (i.e., plasticity). With the development of pre-trained models (PTMs), fine-tuning PTMs on clients has become a promising approach to leveraging their extensive knowledge in FCL. In this paper, we propose MultiFCL, a novel FCL framework that fine-tunes PTMs to adapt to FCL while preserving their strong generalization capabilities. Specifically, to ensure the stability, MultiFCL introduces lightweight adapters for task adaption, which are subsequently frozen to prevent catastrophic forgetting. Moreover, by utilizing the semantic features of old tasks, MultiFCL performs multi-modal initialization of new task class prototypes. To enhance the plasticity, MultiFCL employs a multi-expert training mechanism that integrates multi-scale feature learning with multi-teacher dynamic self-distillation. Through intra-client and inter-client expert communication, MultiFCL facilitates cross-task and cross-client knowledge fusion. Experimental results demonstrate that MultiFCL achieves state-of-the-art performance across multiple datasets and settings, showcasing its effectiveness in FCL scenarios.