Accurate and Efficient Low-Rank Model Merging in Core Space

Joost van de Weijer (Computer Vision Center Barcelona) · Angelo Porrello (University of Modena and Reggio Emilia) · SIMONE CALDERARA (University of Modena and Reggio Emilia, Italy) · Simone Magistri (University of Florence) · Bartłomiej Twardowski (IDEAS NCBR) · Andrew Bagdanov (University of Florence) · Aniello Panariello (University of Modena and Reggio Emilia) · Daniel Marczak (Warsaw University of Technology)

accuracy improvement complexity analysiscore space merging frameworkefficiency preservationempirical results information loss language tasks low-rank adaptationmerging methodsmodel fine-tuningparameter-efficient techniquesprojection state-of-the-art results vision tasks weight matrices

In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging.