Reproduce “Metalearning Continual Learning Algorithms” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
Automated Continual Learning (ACL) trains self-referential neural networks to discover their own in-context continual learning algorithms, directly addressing catastrophic forgetting. The approach achieved superior performance over replay-free hand-crafted and meta-continual learning methods on benchmarks like Split-MNIST.
Key insights
- The paper identified and empirically demonstrated "in-context catastrophic forgetting" (ICCF) in in-context learners without explicit backward transfer objectives.
- Continual learning algorithms can be automatically discovered through meta-learning by framing CL as sequence processing and encoding CL desiderata directly into the loss function.
- Self-referential neural networks can learn to implement their own memory read/write operations to effectively mitigate catastrophic forgetting in a replay-free setting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.