Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learning

Ishani Karmarkar (Stanford University) · Emile Anand (Georgia Institute of Technology and Cognition Labs) · Guannan Qu (Carnegie Mellon University)
action spaceagent interactionsalgorithm efficiencyconvergence boundsdecentralized randomized policyjoint state spacelearning algorithmslocal agent interactionsmean-field theorymulti-agent reinforcement learningoptimal policy convergencepolynomial time complexitysequential global decision-makingsubsample-mfqsubsampled agents

Designing efficient algorithms for multi-agent reinforcement learning (MARL) is fundamentally challenging because the size of the joint state and action spaces grows exponentially in the number of agents. These difficulties are exacerbated when balancing sequential global decision-making with local agent interactions. In this work, we propose a new algorithm $\texttt{SUBSAMPLE-MFQ}$ ($\textbf{Subsample}$-$\textbf{M}$ean-$\textbf{F}$ield-$\textbf{Q}$-learning) and a decentralized randomized policy for a system with $n$ agents. For any $k\leq n$, our algorithm learns a policy for the system in time polynomial in $k$. We prove that this learned policy converges to the optimal policy on the order of $\tilde{O}(1/\sqrt{k})$ as the number of subsampled agents $k$ increases. In particular, this bound is independent of the number of agents $n$.