Reproduce “Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealing” on OpenResearch

About this paper

Bryon Tjanaka, Matthew C. Fontaine, David H. Lee, Aniruddha Kalkar, Stefanos Nikolaidis

Researchers at the University of Southern California introduce scalable variants of the Covariance Matrix Adaptation MAP-Annealing (CMA-MAE) algorithm, enabling it to train diverse, high-dimensional neural network controllers for robotic locomotion tasks. These new variants match or exceed the quality diversity performance of deep reinforcement learning approaches while demonstrating significantly reduced computational time, often operating in less than half the time.

Key insights

• Efficient approximation of the covariance matrix is sufficient for high-dimensional QD problems; increasing search distribution complexity beyond a diagonal matrix does not consistently yield further performance gains for robotic locomotion tasks.
• The soft archive mechanism with an adjustable learning rate (α) is critical for CMA-MAE's performance, effectively balancing objective optimization with measure space exploration to find diverse, high-performing solutions.
• Evolution Strategies offer a computationally efficient and less hyperparameter-dependent alternative to deep RL for high-dimensional Quality Diversity in robotics, maintaining competitive performance.

Summary by alphaXiv.