Deep Reinforcement Learning at the Edge of the Statistical Precipice

Abstraction: Critique of deep RL evaluation using point estimates; proposes robust statistical methodology

Key points:

• Most deep RL results report only mean/median scores across tasks, ignoring statistical uncertainty from few training runs — risking misleading conclusions
• Case study on Atari 100k benchmark shows substantial discrepancies between point-estimate conclusions and thorough statistical analysis
• Proposed remedies: interval estimates of aggregate performance, performance profiles showing full result distributions, and interquartile mean (IQM) scores
• IQM is more robust and efficient than mean/median for small-sample RL evaluation
• Analysis of ALE, Procgen, and DeepMind Control Suite reveals prior comparison discrepancies
• Introduces rliable, an open-source library for reliable RL evaluation

Connections: Rliable · Reinforcement Learning · Statistical Evaluation · Benchmarking

Source: https://arxiv.org/abs/2108.13264