Reproduce “Effective Diversity in Population Based Reinforcement Learning” on OpenResearch

About this paper

Jack Parker-Holder, Aldo Pacchiano, Krzysztof Choromanski, Stephen Roberts

Researchers from Oxford, UC Berkeley, and Google Brain Robotics developed Diversity via Determinants (DvD), a framework that enhances exploration in population-based reinforcement learning by measuring diversity using the determinant of a kernel matrix of task-agnostic behavioral embeddings. The method achieves improved exploration on challenging tasks, including state-of-the-art performance and sample efficiency on Humanoid-v2, while effectively discovering multi-modal solutions.

Key insights

• Determinant-based diversity measures capture the 'volume' spanned by policy behaviors, effectively penalizing redundancy and promoting truly distinct solutions, unlike pairwise distance metrics.
• Jointly optimizing the entire population simultaneously prevents individual agents from repeatedly cycling between similar behaviors, leading to more coherent and extensive exploration.
• Adaptive adjustment of the reward-diversity trade-off allows the algorithm to dynamically prioritize exploration or exploitation based on learning progress, improving robustness and efficiency.

Summary by alphaXiv.