Reproduce “Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning” on OpenResearch

About this paper

Ke Sun, Hongming Zhang, Jun Jin, Chao Gao, Xi Chen, Wulong Liu, Linglong Kong

Researchers from the University of Alberta and collaborators developed FAME, a dual-learner framework for Continual Reinforcement Learning, which introduces formal definitions for MDP distance and catastrophic forgetting. Inspired by human hippocampal-cortical memory, FAME employs a fast learner for rapid adaptation and a meta learner for knowledge integration, outperforming baselines in average performance, forward transfer, and forgetting across pixel-based and continuous control tasks.

Key insights

• A principled approach to Continual RL benefits from formal definitions of MDP similarity and catastrophic forgetting, enabling clearer optimization targets for algorithmic design and analysis.
• A dual-learner architecture, inspired by the human brain's memory system, effectively addresses the trade-off between plasticity (rapid adaptation to new tasks) and stability (retention of old knowledge).
• Adaptive knowledge transfer mechanisms, such as intelligently selecting initial policies (e.g., from a meta-learner or randomly), are crucial to leverage prior knowledge effectively and avoid negative transfer in sequential learning.

Summary by alphaXiv.