Beyond Expectations: Quantile-Guided Alignment for Risk-Calibrated Language Models

Xinran Wang (Beijing University of Posts and Telecommunications) · Jiawei Zhou (Stony Brook University) · Jin Du (University of Minnesota - Twin Cities) · Azal Khan (University of Minnesota - Twin Cities) · qi le (University of Minnesota) · Enmao Diao (ColAI) · Jie Ding (University of Minnesota) · Ali Anwar (University of Minnesota)

augmented reward formulationcatastrophic outputscode-generation tasksconversation tasksdesired improvementsdistribution of outputshigh-risk tail eventsqaquantile alignmentquantile constraintsquantile-guided alignmentreinforcement learning from human feedbackrisk-calibrated language modelsrlhftail-focused alignment

Large language models can generate rare but catastrophic outputs, such as harmful conversations or insecure code. Existing Reinforcement Learning from Human Feedback (RLHF) typically maximizes average reward, leaving high-risk tail events insufficiently controlled. We introduce Quantile‑Guided Alignment (QA), a framework that allows users to specify desired improvements at any quantile—individually or across multiple reward dimensions—thus shifting the distribution of outputs with finer control toward safer, more desirable outcomes. The method extends standard RLHF via an augmented reward formulation that enforces quantile constraints. Experiments on conversation and code‐generation tasks show that quantile alignment significantly enhances quality at targeted tails while maintaining overall performance. The results position QA as a principled route to risk‑calibrated language models with tail‑focused alignment.