Reproduce “Curiosity-driven Exploration in Sparse-reward Multi-agent Reinforcement Learning” on OpenResearch

About this paper

Jiong Li, Pratik Gajane

This research introduces I-Go-Explore, a method that integrates Go-Explore principles with the Intrinsic Curiosity Module (ICM) to enhance exploration in sparse-reward multi-agent reinforcement learning environments. The approach improves the prey agent's average reward by over 50% and reduces training time by approximately 12% compared to baseline ICM in long-term multi-agent predator-prey scenarios.

Key insights

• Integrating a Go-Explore-inspired state archiving mechanism with intrinsic curiosity can effectively address the detachment problem in sparse-reward multi-agent settings.
• A structured exploration phase, where agents revisit and build upon previously achieved states, significantly improves sample efficiency and long-term learning performance over purely curiosity-driven methods.
• The optimal length of the additional exploration phase is critical; excessive arbitrary exploration can negatively impact the learned policy, suggesting a need for careful tuning.

Summary by alphaXiv.