255

rlhfai-safetyalignmentopenai-history

Abstraction: Paul Christiano's defense of RLHF research positive net impact

Key points:

Paul Christiano led RLHF work at OpenAI 2017-2020; started with human feedback because synthetic feedback was less compelling to skeptics
Argues RLHF was net positive: it implemented a foundational alignment step, motivated harder problems, and enabled in-vivo study of reward hacking and deceptive alignment
Rejects the claim that RLHF "covers up problems" — models without RLHF still expose issues via other methods, and RLHF produces more realistic failure modes to study
ChatGPT's success over GPT-3 debated: Christiano attributes more to chat interface and deployment framing than to RLHF alone; others argue RLHF was the key quality differentiator
Estimates his RLHF work accelerated AI progress by a few tenths of a percent, cutting timelines by roughly a few days — not negligible but not a dominant factor
Future work should focus on robustness to out-of-distribution failures, domains where overseers cannot judge results, and in-vitro examples of treacherous turns

Connections: Paul Christiano · Openai · Chatgpt · Reinforcement Learning From Human Feedback · AI Safety