Reproduce “Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xutong Zhao, Yaqi Xie

Xutong Zhao and Yaqi Xie from Mila introduce Multi-level Advantage Credit Assignment (MACA), an actor-critic method for cooperative Multi-Agent Reinforcement Learning (MARL) that formalizes and integrates credit assignment at individual, joint, and dynamically correlated agent subset levels using an attention mechanism. MACA achieves superior win rates and higher sample efficiency on StarCraft Multi-Agent Challenge (SMAC) benchmarks, outperforming state-of-the-art methods in complex, stochastic environments.

Key insights

  • Cooperative rewards in MARL can arise from dynamically varying 'levels' of agent collaboration (individual, subsets, full group), necessitating a multi-level credit assignment approach.
  • Attention mechanisms can effectively identify context-dependent agent correlations, enabling the formation of dynamic 'CorrSets' for more accurate credit attribution to specific coalitions.
  • Integrating counterfactual advantages from individual, joint, and dynamically correlated agent actions provides a robust and comprehensive signal for policy optimization in complex cooperative tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.