FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts

Xiangyang Ji (Tsinghua University) · Heming Zou (Tsinghua University) · Yunliang Zang (Tianjin University) · Wutong Xu (University of Maryland, College Park) · Yao Zhu (Zhejiang University)

ai technologiesbiological structurescomputational efficiencydown-projectionexpert routingimplicit routerinter-task interferenceintra-task correlationslow-rank adaptation mixture-of-expertsmulti-task model mergingorthogonality propertyparameter-efficient fine-tuningrank-wise expert activationsparse random projection

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for foundation models, but it suffers from parameter interference, resulting in suboptimal performance. Although Mixture-of-Experts (MoE)-based LoRA variants show promise in mitigating intra-task correlations in single-task instruction tuning, they introduce additional router parameters and remain ineffective in multi-task model merging where inter-task interference arises. Inspired by the fly olfactory circuit, we propose FlyLoRA, an implicit MoE-based LoRA variant that introduces: (1) rank-wise expert activation in the up-projection matrix, and (2) an implicit router that unifies expert routing and down-projection, where a frozen sparse random projection matrix replaces the traditional dense trainable version. This design resolves the trade-off between intra-task decorrelation and computational efficiency by eliminating the need for an explicit router, while inherently mitigating inter-task interference due to the orthogonality property of random matrices. Extensive experiments across four domains