Reproduce “On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Andrew Cohen, Ervin Teng, Vincent-Pierre Berges, Ruo-Ping Dong, Hunter Henry, Marwan Mattar, Alexander Zook, Sujoy Ganguly

Researchers from Unity Technologies developed MA-POCA, a multi-agent reinforcement learning architecture that uses self-attention mechanisms to effectively manage dynamic agent populations, avoiding the inefficiencies of absorbing states and robustly addressing the Posthumous Credit Assignment Problem. This approach enables faster and more stable learning in environments where agents can dynamically appear and disappear, outperforming conventional methods in novel dynamic tasks.

Key insights

  • Absorbing states significantly increase the sample complexity and reduce the robustness of neural networks in handling variable-length inputs, as demonstrated by a toy supervised learning task.
  • Self-attention mechanisms effectively address the variable input dimension problem and improve learning efficiency by processing only active agents and capturing their interdependencies.
  • Properly attributing rewards to agents for actions taken before their termination (Posthumous Credit Assignment) is crucial for learning optimal policies, especially sacrificial strategies, in dynamic multi-agent environments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.