The challenges of reinforcement learning from human feedback (RLHF) - TechTalks

rlhfalignmentreward-hackingllm-safety

Abstraction: RLHF limitations across feedback, reward modeling, and policy

Key points:

RLHF consists of three components: feedback collection, reward modeling, and policy optimization — each with distinct failure modes
Human annotators suffer from fatigue, bias, and inability to evaluate complex tasks (e.g., code security vulnerabilities)
Reward hacking lets models exploit shortcuts that score well on the reward model without genuinely aligning with human values
RL fine-tuning causes mode collapse: models over-generate high-scoring outputs, reducing diversity and creativity
RLHF models are susceptible to adversarial attacks even when trained with near-perfect reward signals
Proposed mitigations include ensemble reward models, pre-training data filtering by human feedback, and multi-pronged safety measures rather than treating RLHF as a complete solution

Connections: Openai · Chatgpt · Reinforcement Learning From Human Feedback · AI Safety