Reproduce “Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sebastian Lee, Stefano Sarao Mannelli, Claudia Clopath, Sebastian Goldt, Andrew Saxe
The paper explains intermediate-similarity catastrophic forgetting through a proposed trade-off between reusing hidden units tuned to an earlier task and activating quiet units. It analyzes a two-layer teacher–student model with training dynamics and checks the pattern on Fashion-MNIST. Restarting before interleaved training improves first-task forgetting and second-task performance in the intermediate regime, but this account is not established as universal.
Key insights
- Catastrophic forgetting in the intermediate task similarity regime results from a detrimental compromise where the network attempts to re-use partially relevant specialized nodes, leading to interference, rather than effectively dedicating new resources.
- Elastic Weight Consolidation (EWC) reduces forgetting by biasing networks towards activating new nodes or freezing existing weights, being particularly effective for tasks with intermediate similarity.
- Interleaved replay struggles to fully alleviate forgetting in the intermediate similarity regime and can lead to "catastrophic slowing," where re-initializing the network is sometimes more beneficial than continuing from a pre-trained state.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.