Deep Reinforcement Learning Doesn't Work Yet

Abstraction: Systematic critique of deep RL limitations and failure modes

Key points:

• Sample inefficiency: Rainbow DQN needs ~18M Atari frames (~83 hours) to reach human-level median performance; MuJoCo tasks require 10^5–10^7 steps even without vision
• Domain-specific methods (MPC, MCTS with ground-truth model) consistently outperform model-free deep RL when final performance is the goal
• Reward misspecification causes reward hacking — RL exploits shaped rewards in unintended ways (boat racing powerup farming, block flipping instead of stacking, table-slamming)
• Training instability: ~30% failure rate across random seeds on simple tasks is considered acceptable; different implementations of the same algorithm produce different results
• Deep RL most likely to succeed when: experience is abundant, problem is simplified, self-play is possible, reward is clean and ungameable
• Neural Architecture Search succeeds partly because validation accuracy is a rich, legitimate reward signal and hyperparameters are nearly linearly independent

Connections: Deepmind · Openai · Google Brain · Reinforcement Learning · Reward Function Design · Exploration Exploitation

Source: https://www.alexirpan.com/2018/02/14/rl-hard.html