Decisions from Data: How Offline Reinforcement Learning Will Change How We Use ML

Abstraction: Argument for offline RL enabling data-driven end-to-end decision making

Key points:

• Distinguishes supervised prediction (predicts quantities, requires manual decision rules, assumes i.i.d. data) from sequential decision making (specifies objectives, outputs optimal actions, handles feedback and non-i.i.d. data)
• Offline RL (also called batch RL) learns policies from a fixed pre-collected dataset with no further environment interaction, analogous to how supervised learning uses static datasets
• Core technical challenge is distributional shift: the Q-function is trained under the behavior policy's distribution but queried under the learned policy, leading to adversarial out-of-distribution action exploitation
• Three key approaches: policy constraints (KL or support divergence from behavior policy), implicit constraints (AWR/AWAC via weighted maximum likelihood), and conservative Q-functions (CQL — regularizes Q-values for OOD actions downward, achieves SOTA without explicit policy constraint)
• Offline RL can leverage large diverse datasets (analogous to ImageNet scale) for decision problems in autonomous vehicles, HVAC control, traffic optimization
• Can be combined with modest online fine-tuning (AWAC) after initial offline phase

Connections: Sergey Levine · Offline Reinforcement Learning · Reinforcement Learning · Distributional Shift

Source: https://medium.com/@sergey.levine/decisions-from-data-how-offline-reinforcement-learning-will-change-how-we-use-ml-24d98cb069b0