Reproduce “Learning Progress Driven Multi-Agent Curriculum” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenshuai Zhao, Zhiyuan Li, Joni Pajarinen

Researchers from Aalto University developed SPMARL, an adaptive curriculum learning method for Multi-Agent Reinforcement Learning that uses Temporal Difference (TD) error as a learning progress objective to dynamically adjust the number of agents. This approach led to faster convergence and superior final performance compared to existing baselines across various sparse-reward MARL tasks.

Key insights

  • Temporal Difference (TD) error provides a significantly more stable and reliable measure of learning progress in MARL compared to sparse episode returns, which suffer from high variance.
  • Optimizing for tasks that maximize TD error effectively directs the curriculum to environments where agents are undergoing substantial policy changes, leading to more efficient learning.
  • The number of agents is a powerful curriculum variable in MARL, and dynamically adapting this count based on a robust learning progress measure effectively mitigates exploration challenges in sparse-reward settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.