Reproduce “How Exploration Breaks Cooperation in Shared-Policy Multi-Agent Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi-Ning Weng, Hsuan-Wei Lee
This research systematically characterizes how increasing exploration in shared-policy Deep Q-Networks (DQNs) leads to a breakdown of cooperation in dynamic multi-agent environments. The study demonstrates that this failure mode arises from a representational bottleneck within the shared network, causing a degradation of action-value discrimination rather than unstable learning.
Key insights
- Cooperation collapse in shared-policy DQNs is a structural failure stemming from a representational bottleneck, where the shared network struggles to maintain action-discriminative features under aggregated non-stationarity caused by exploration.
- The system reliably converges to stable, but low-cooperation, solutions due to a systematic erosion of value discrimination, rather than training instability or divergent behavior.
- Partial observability of learning dynamics significantly amplifies this representational bottleneck; augmenting the state with temporal or training-progress signals can delay the cooperation collapse.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.