Reproduce “Why Do Neural Networks Forget: A Study of Collapse in Continual Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yunqin Zhu, Jun Jin

Effective-rank analysis is used to connect catastrophic forgetting with the gradual collapse of neural networks’ weight and activation spaces during continual learning. Across Split MNIST and Split CIFAR-100, declining effective rank tracks worsening retention, while Experience Replay preserves both accuracy and internal dimensionality more reliably than Learning without Forgetting.

Key insights

  • Forgetting is accompanied by a narrowing of both hidden-feature and parameter subspaces: lower effective rank generally coincides with reduced plasticity and poorer retention of earlier tasks.
  • Preserving model outputs is not sufficient to preserve internal capacity. Learning without Forgetting can delay performance loss, but it does not consistently prevent weight-rank collapse.
  • Experience Replay helps maintain old-task decision boundaries while continuing to support diverse feature directions, making it more effective than output distillation in the reported experiments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.