Reproduce “Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Naoki Shitanda, Motoki Omura, Tatsuya Harada, Takayuki Osa

Coupled Policy Optimization (CPO) introduces a method to regulate inter-policy diversity in ensemble reinforcement learning for massively parallel environments, demonstrating improved sample efficiency and stable learning. CPO consistently achieves higher performance and faster convergence across complex robotic manipulation tasks by controlling follower policies' divergence from the leader via KL constraints and an adversarial reward.

Key insights

  • Excessive inter-policy divergence inversely relates to the Effective Sample Size (ESS) and directly correlates with increased PPO clipping bias, degrading learning stability and sample efficiency.
  • KL divergence effectively upper-bounds the expected absolute deviation of Importance Sampling (IS) ratios from 1, providing a theoretical basis for regulating inter-policy distance.
  • Maintaining stable inter-agent distances and structured policy formation, where followers are distributed around the leader, generates more informative samples and leads to robust learning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.