Whose Instructions Count? Resolving Preference Bias in Instruction Fine-Tuning

Weihao Luo (Donghua University, Shanghai) · Xuan Zhang (Texas A&M University) · Jiayu Zhang (Westlake University) · Changbang Li (University of Pennsylvania) · Yinan Peng (Hengxin Technology Ltd.) · Peilai Yu (University of Munich, Ludwig-Maximilians-Universität München)
accuracy/f1-emcross-layer prefix alignmentdynamic cross-layer preference correctiongeneralizationhidden statesinstruction fine-tuninginter-seed variancepreference biaspreference correction modulepreference-sensitive similarity estimatorpreference-shifted variantsrobustnesssemantic representationsstate of the artsuper/glue tasks

Instruction fine-tuning (IFT) has emerged as a ubiquitous strategy for specializing large language models (LLMs), yet it implicitly assumes a single, coherent "ground-truth" preference behind all human-written instructions. In practice, annotators differ in the styles, emphases, and granularities they prefer, introducing preference bias that can erode both robustness and generalization. We propose Dynamic Cross-Layer Preference Correction (\textsc{DCPC}), it couples (i) a preference-sensitive similarity estimator that detects mismatched instructional cues, (ii) cross-layer prefix alignment to reconcile semantic representations across transformer layers, and (iii) a lightweight Preference Correction Module (PCM) that dynamically adjusts hidden states to honor the inferred dominant preference. On five Super/GLUE tasks and the Alpaca set—plus six preference-shifted variants—DCPC boosts accuracy/F1-EM by 4.0–6.7 points and gpt-score by +0.7, while cutting inter-seed variance up to 35% on LlaMA-2 13B and Mistral-7B, setting a new state of the art for robust instruction tuning.