C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models

Yucheng Wang (Texas A&M University - College Station) · Byung-Jun Yoon (Texas A&M University) · Nathan Urban (Brookhaven National Laboratory) · Xiaoning Qian (Texas A&M) · Amir Hossein Rahmati (Texas A&M University - College Station) · Sanket Jantre (Brookhaven National Laboratory) · Weifeng Zhang (Texas A&M University)
ablation studiesarchitecture-agnosticcontextual low-rank adaptationfew-shot settingsfine-tuninglightweight moduleslow-rank adaptationmodel generalizationoverfittingparameter posteriorspredictive uncertaintyrobust predictionsuncertainty quantificationuncertainty-awarewell-calibrated uncertainties

Low-Rank Adaptation (LoRA) offers a cost-effective solution for fine-tuning large language models (LLMs), but it often produces overconfident predictions in data-scarce few-shot settings. To address this issue, several classical statistical learning approaches have been repurposed for scalable uncertainty-aware LoRA fine-tuning. However, these approaches neglect how input characteristics affect the predictive uncertainty estimates. To address this limitation, we propose Contextual Low-Rank Adaptation (**C-LoRA**) as a novel uncertainty-aware and parameter efficient fine-tuning approach, by developing new lightweight LoRA modules contextualized to each input data sample to dynamically adapt uncertainty estimates. Incorporating data-driven contexts into the parameter posteriors, C-LoRA mitigates overfitting, achieves well-calibrated uncertainties, and yields robust predictions. Extensive experiments on LLaMA2-7B models demonstrate that C-LoRA consistently outperforms the state-of-the-art uncertainty-aware LoRA methods in both uncertainty quantification and model generalization. Ablation studies further confirm the critical role of our contextual modules in capturing sample-specific uncertainties. C-LoRA sets a new standard for robust, uncertainty-aware LLM fine-tuning in few-shot regimes. Although our experiments are limited to 7B models, our method is architecture-agnostic and, in principle, applies beyond this scale; studying its scaling to larger models remains an open problem. Our code is available at https://github.com/ahra99/c_lora.