Reproduce “Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hongyao Tang, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Glen Berseth

Researchers from Mila demonstrate that plasticity loss in continual reinforcement learning is linked to increased network churn, manifesting as a decrease in the Neural Tangent Kernel's rank. They introduce C-CHAIN, a method that mitigates this churn, leading to improved learning performance and robustness across diverse continual RL environments, including OpenAI Gym, ProcGen, DeepMind Control, and MinAtar.

Key insights

  • Plasticity loss in continual RL is directly associated with an exacerbation of network churn, evidenced by the decrease in the NTK matrix rank and increased gradient correlations.
  • C-CHAIN mitigates plasticity loss through a "gradient decorrelation effect" (suppressing off-diagonal NTK entries) and a "step-size adjustment effect," with gradient decorrelation being the more dominant factor.
  • The benefits of churn reduction are more pronounced in continual RL settings compared to continual supervised learning, suggesting fundamental differences in how plasticity loss manifests in these paradigms.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.