Reproduce “Learning Cooperative Multi-Agent Policies with Partial Reward Decoupling” on OpenResearch

About this paper

Benjamin Freed, Aditya Kapoor, Ian Abraham, Jeff Schneider, Howie Choset

Researchers from Carnegie Mellon University and Yale University developed Partial Reward Decoupling (PRD), a novel approach that dynamically decomposes multi-agent cooperative problems to address the credit assignment challenge. This method, integrated into an Actor-Critic framework, improves data efficiency, learning stability, and asymptotic performance by reducing policy gradient variance and effectively utilizing individual reward streams.

Key insights

• An agent's policy gradient can be optimized by considering only the subset of other agents whose future rewards are causally impacted by its actions, allowing for partial reward decoupling without introducing bias.
• Dynamically identifying these causally linked subgroups using a graph attention network within a centralized critic helps reduce the variance of policy gradient estimates.
• This approach allows for the effective utilization of individual reward streams and compatibility with advanced advantage estimation techniques like GAE, which existing credit assignment methods often lack.

Summary by alphaXiv.