Reproduce “Self-Activating Neural Ensembles for Continual Reinforcement Learning” on OpenResearch

About this paper

Sam Powers, Eliot Xing, Abhinav Gupta

Self-Activating Neural Ensembles (SANE) introduces a dynamic, modular architectural approach to continual reinforcement learning that autonomously mitigates catastrophic forgetting without requiring explicit task boundaries or human supervision. The system achieves superior performance on challenging Procgen environments by dynamically creating and merging specialized neural modules, protecting previously acquired skills.

Key insights

• Task-agnostic continual learning is achievable by enabling agents to autonomously detect changes in their environment and dynamically allocate specialized neural resources.
• Modular ensembles, when dynamically managed and self-activated, provide an effective architectural solution to mitigate catastrophic forgetting in reinforcement learning.
• The concept of 'drift' detected by anchor critics, combined with uncertainty estimates, allows for robust, data-driven decisions on when to create new knowledge specialists or merge existing ones.

Summary by alphaXiv.