Reproduce “Learning to Forget: Continual Learning with Adaptive Weight Decay” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Aditya A. Ramesh, Alex Lewandowski, Jürgen Schmidhuber

Forgetting through Adaptive DEcay (FADE) automates the management of knowledge retention in neural networks by dynamically adjusting individual parameter decay rates, demonstrating enhanced learning performance and stability in online continual learning scenarios.

Key insights

  • Controlled and adaptive forgetting, rather than just retention, is essential in continual learning to free up model capacity and effectively manage the stability-plasticity trade-off.
  • Per-parameter adaptive weight decay allows a model to automatically differentiate between relevant and irrelevant information, selectively retaining stable knowledge while discarding outdated parameters.
  • Adaptive decay rates complement adaptive learning rates, suggesting that optimally managing both how fast new information is integrated and how much old information is retained can lead to better continual learning performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.