RidgeLoRA: Matrix Ridge Enhanced Low-Rank Adaptation of Large Language Models

Lifeng Shang (Huawei Technologies Ltd.) · Yasheng Wang (Huawei Technologies Ltd.) · Junda Zhu (Beihang University) · Jun Ai (Beihang University) · Yujun Li (Huawei Technologies Ltd.) · Yichun Yin (Huawei Noah's Ark Lab) · Qun Liu (Huawei Noah's Ark Lab)

computational cost extensive experimentsfull-rank approximationlightweight architecturelow-rank adaptationmatrix ridgemodel optimization parameter-efficient fine-tuning performance comparisonrepresentation abilityridgelorarigorous mathematical derivationsuboptimal performancetrainable low-rank matrixupper bound

As one of the state-of-the-art parameter-efficient fine-tuning~(PEFT) methods, Low-Rank Adaptation (LoRA) enables model optimization with reduced computational cost through trainable low-rank matrix. However, the low-rank nature makes it prone to produce a decrease in the representation ability, leading to suboptimal performance. In order to break this limitation, we propose RidgeLoRA, a lightweight architecture like LoRA that incorporates novel architecture and matrix ridge enhanced full-rank approximation, to match the performance of full-rank training, while eliminating the need for high memory and a large number of parameters to restore the rank of matrices. We provide a rigorous mathematical derivation to prove that RidgeLoRA has a better upper bound on the representations than vanilla LoRA. Furthermore, extensive experiments across multiple domains demonstrate that RidgeLoRA achieves better performance than other LoRA variants, and can even match or surpass full-rank training.