Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning

Yaodong Yang (AIG) · Bo An (Nanyang Technological University) · Chengdong Ma · Simin Li (Beihang University) · Zihao Mao (Beijing University of Aeronautics and Astronautics) · Hanxiao Li (Beihang University) · Zonglei Jing (Beijing University of Aeronautics and Astronautics) · Zhuohang bian (Beijing University of Aeronautics and Astronautics) · Jun Guo (Tsinghua University) · Li Wang (Beijing Institute of Technology) · Zhuoran Han (Beijing University of Aeronautics and Astronautics) · Ruixiao Xu (Beijing University of Aeronautics and Astronautics) · Xin Yu (Institute of automation, Chinese academy of science) · Yuqing Ma (Beijing University of Aeronautics and Astronautics) · Weifeng Lv · Xianglong Liu (Beihang University, Tsinghua University)

action noisecooperative performancecritic learning ratesearly stopping empirical studygeneralized advantage estimationhyperparameter tuningleaky relumulti-agent reinforcement learningobservation noiseparameter sharingpopartresilience robustnessuncertainty types

In cooperative Multi-Agent Reinforcement Learning (MARL), it is a common practice to tune hyperparameters in ideal simulated environments to maximize cooperative performance. However, policies tuned for cooperation often fail to maintain robustness and resilience under real-world uncertainties. Building trustworthy MARL systems requires a deep understanding of \emph{robustness}, which ensures stability under uncertainties, and \emph{resilience}, the ability to recover from disruptions—a concept extensively studied in control systems but largely overlooked in MARL. In this paper, we present a large-scale empirical study comprising over 82,620 experiments to evaluate cooperation, robustness, and resilience in MARL across 4 real-world environments, 13 uncertainty types, and 15 hyperparameters. Our key findings are: (1) Under mild uncertainty, optimizing cooperation improves robustness and resilience, but this link weakens as perturbations intensify. Robustness and resilience also varies by algorithm and uncertainty type. (2) Robustness and resilience do not generalize across uncertainty modalities or agent scopes: policies robust to action noise for all agents may fail under observation noise on a single agent. (3) Hyperparameter tuning is critical for trustworthy MARL: surprisingly, standard practices like parameter sharing, GAE, and PopArt can hurt robustness, while early stopping, high critic learning rates, and Leaky ReLU consistently help. By optimizing hyperparameters only, we observe substantial improvement in cooperation, robustness and resilience across all MARL backbones, with the phenomenon also generalizing to robust MARL methods across these backbones.