Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
alignment methodsaverage utilitybradley-terry modelscomparison pairsdistortiondpokl divergenceslearned policyminimax optimal distortionnash learning from human feedbackpairwise comparisonspluralistic alignmentrlhfsocial choice theoryutility distributions
After pre-training, large language models are aligned with human preferences based on pairwise comparisons. State-of-the-art alignment methods (such as PPO-based RLHF and DPO) are built on the assumption of aligning with a single preference model, despite being deployed in settings where users have diverse preferences. As a result, it is not even clear that these alignment methods produce models that satisfy users \emph{on average}