Reproduce “Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuchen Xiao, Weihao Tan, Christopher Amato

This research introduces asynchronous multi-agent actor-critic methods designed for scenarios where agents execute temporally-extended macro-actions of varying durations. The Mac-IAICC method, featuring individual centralized critics, achieved superior performance and scalability in complex cooperative tasks compared to existing synchronous and naive asynchronous baselines, with learned policies successfully deployed on real robots.

Key insights

  • Learning with temporally-extended macro-actions substantially improves performance and scalability in complex multi-agent tasks, making previously intractable problems solvable and facilitating long-horizon reasoning.
  • Individualized centralized critics (Mac-IAICC) are crucial for robust and scalable Centralized Training for Decentralized Execution (CTDE) in asynchronous multi-agent settings, effectively mitigating reward assignment and variance issues of shared critics.
  • Actor-critic methods demonstrate better scalability for large joint macro-action spaces and improved exploration properties compared to value-based macro-action approaches in complex multi-agent environments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.