Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Observation Delays

Songchen Fu (Institute of Acoustics, Chinese Academy of Sciences) · Siang Chen (Tsinghua University) · Shaojing Zhao (University of the Chinese Academy of Sciences) · Letian Bai (University of the Chinese Academy of Sciences) · Hong Liang (University of the Chinese Academy of Sciences) · Ta Li (Institute of Acoustics, Chinese Academy of Sciences) · Yonghong Yan (Institute of Acoustics, Chinese Academy of Sciences)

dec-pomdpdecentralized stochastic individual delay partially observable markov decision processdelay-free performancedelayed observation problemsgeneralizabilityimplementation modulesmarl benchmarks multi-agent reinforcement learning multi-agent systemsobservation delaysobservation generation patternperformance degradationrainbow delay compensationsolution frameworkstochastic individual delays

In real-world multi-agent systems (MASs), observation delays are ubiquitous, preventing agents from making decisions based on the environment's true state. An individual agent's local observation typically comprises multiple components from other agents or dynamic entities within the environment. These discrete observation components with varying delay characteristics pose significant challenges for multi-agent reinforcement learning (MARL). In this paper, we first formulate the decentralized stochastic individual delay partially observable Markov decision process (DSID-POMDP) by extending the standard Dec-POMDP. We then propose the Rainbow Delay Compensation (RDC), a MARL training framework for addressing stochastic individual delays, along with recommended implementations for its constituent modules. We implement the DSID-POMDP's observation generation pattern using standard MARL benchmarks, including MPE and SMAC. Experiments demonstrate that baseline MARL methods suffer severe performance degradation under fixed and unfixed delays. The RDC-enhanced approach mitigates this issue, remarkably achieving ideal delay-free performance in certain delay scenarios while maintaining generalizability. Our work provides a novel perspective on multi-agent delayed observation problems and offers an effective solution framework. The source code is available at https://github.com/linkjoker1006/RDC-pymarl.