Reproduce “Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning” on OpenResearch

About this paper

Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, Wei Pan

Researchers at University College London developed the Probabilistic Recursive Reasoning (PR2) framework, which enables multi-agent reinforcement learning agents to anticipate how opponents would react to their own actions. This approach leads to improved learning stability and convergence to Nash equilibria in complex multi-agent games where traditional gradient-based methods typically fail or cycle, while maintaining decentralized training and execution.

Key insights

• Explicitly modeling opponents' conditional reactions to an agent's future actions (level-1 recursive reasoning) is crucial for stable convergence in complex multi-agent games, overcoming limitations of non-correlated policy assumptions.
• Variational inference and soft reinforcement learning provide a principled way to approximate the optimal opponent conditional policy from an agent's own Q-function, enabling decentralized recursive reasoning.
• PR2's approach leads to algorithms that can converge to unique Nash equilibria in scenarios where common gradient-based MARL methods exhibit cycling or converge to suboptimal local optima.
• The framework allows for decentralized training and execution, making it scalable and applicable to settings where centralized control or full observability is not feasible.
• The theoretical guarantee that the PR2 soft value iteration operator is a contraction mapping ensures convergence for symmetric games with a unique equilibrium under self-play, demonstrating the method's stability and robustness.

Summary by alphaXiv.