Reproduce “Coordinated Proximal Policy Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zifan Wu, Chao Yu, Deheng Ye, Junge Zhang, Haiyin Piao, Hankz Hankui Zhuo
Coordinated Proximal Policy Optimization (CoPPO) introduces a policy-based algorithm that extends PPO for cooperative multi-agent reinforcement learning by coordinating policy updates across agents. It demonstrates superior performance and stability against other policy-based methods in cooperative matrix games and achieves competitive results with state-of-the-art approaches in the challenging StarCraft II micromanagement environment.
Key insights
- Coordinated step size adaptation across agents, based on their policy ratios, enhances learning stability and performance in cooperative multi-agent settings.
- Dynamic credit assignment can be effectively achieved by modulating an agent's advantage with the clipped product of other agents' policy ratios, allowing agents to 'rethink' actions based on teammates' updates.
- Policy-based multi-agent reinforcement learning methods can achieve performance competitive with or superior to value-based approaches when non-stationarity and variance are appropriately addressed.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.