Reproduce “Deep Coordination Graphs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wendelin Böhmer, Vitaly Kurin, Shimon Whiteson

Deep Coordination Graph (DCG) introduces a scalable deep reinforcement learning algorithm that integrates coordination graphs with deep neural networks to address relative overgeneralization in cooperative multi-agent tasks. It effectively scales to large state and action spaces, achieving robust learning in predator-prey environments and competitive performance on StarCraft II micromanagement benchmarks.

Key insights

  • Integrating coordination graphs with deep neural networks and extensive parameter sharing allows for scaling to large state and action spaces while maintaining sample efficiency.
  • Message passing on the coordination graph provides a computationally efficient mechanism for maximizing the factored joint Q-value function, which is crucial for practical application with many agents.
  • Low-rank approximations for payoff matrices are essential for applying coordination graphs to environments with large action spaces, like StarCraft II, by reducing parameter count and accelerating learning.
  • Relative overgeneralization is a significant pathology in decentralized MARL that can be successfully mitigated by richer value factorizations capable of modeling pairwise agent dependencies, as demonstrated by DCG.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.