Correlated Low-Rank Adaptation for ConvNets

Weijia Zhang (Shanghai Jiao Tong University) · Zhu (Guangdong OPPO Mobile Telecommunications Corp.,Ltd.) · Chao Ma (Microsoft Research Cambridge) · Wu Ran (Shanghai Jiaotong University) · ShuYang Pang (CISDI Group Co., Ltd) · Jinfan Liu (Shanghai Jiaotong University) · JingSheng Liu (CISDI Group Co., Ltd) · Xin Cao (CISDI Group Co., Ltd) · Qiang Li (Huazhong University of Science and Technology) · Yichao Yan (Shanghai Jiao Tong University)

convnext-sconvolutional networkscorrelated low-rank adaptationimage classificationinter-layer dependencieslow-rank adaptationnon-informative local regionsobject detection parameter-efficient fine-tuningparameter-free filteringreceptive field semantic segmentation state-of-the-art approachestuning efficiencyvtab-1k dataset

Low-Rank Adaptation (LoRA) methods have demonstrated considerable success in achieving parameter-efficient fine-tuning (PEFT) for Transformer-based foundation models. These methods typically fine-tune individual Transformer layers using independent LoRA adaptations. However, directly applying existing LoRA techniques to convolutional networks (ConvNets) yields unsatisfactory results due to the high correlation between the stacked sequential layers of ConvNets. To overcome this challenge, we introduce a novel framework called Correlated Low-Rank Adaptation (CoLoRA), which explicitly utilizes correlated low-rank matrices to model the inter-layer dependencies among convolutional layers. Additionally, to enhance tuning efficiency, we propose a parameter-free filtering method that enlarges the receptive field of LoRA, thus minimizing interference from non-informative local regions. Comprehensive experiments conducted across various mainstream vision tasks, including image classification, semantic segmentation, and object detection, illustrate that CoLoRA significantly advances the state-of-the-art PEFT approaches. Notably, our CoLoRA achieves superior performance with only 5\% of trainable parameters, surpassing full fine-tuning in the image classification task on the VTAB-1k dataset using ConvNeXt-S. Code is available at [https://github.com/VISION-SJTU/CoLoRA](https://github.com/VISION-SJTU/CoLoRA).