Reproduce “Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Dan Qiao, Wenhao Li, Shanchao Yang, Hongyuan Zha, Baoxiang Wang

OMSD, a framework for offline Multi-Agent Reinforcement Learning, enables learning coordinated policies from multimodal datasets by using sequential decomposition and diffusion models for conditional score regularization. The method demonstrated superior performance on multi-agent continuous control tasks, achieving up to 107.5% improvement on MPE tasks and an average 73.9% improvement on MaMuJoCo OMIGA datasets over baselines, particularly on medium and random quality data.

Key insights

  • Independent regularization of agent policies in multimodal offline datasets causes "combinatorial mode shift," driving joint policies to incoherent, out-of-distribution regions.
  • Sequential decomposition provides a more accurate, unbiased reference for regularization by accounting for inter-agent conditional dependencies, guiding policies toward in-distribution cooperative behaviors.
  • Diffusion models are effective for estimating complex conditional score functions, which is crucial for robustly navigating and regularizing policies within multimodal behavior distributions present in offline data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.