Reproduce “Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization” on OpenResearch

About this paper

Nicolas Y. Masse, Gregory D. Grant, David J. Freedman

Researchers at The University of Chicago developed a method combining Context-dependent Gating (XdG) with synaptic stabilization to mitigate catastrophic forgetting in neural networks. This approach enabled networks to maintain high performance across hundreds of sequential tasks, achieving up to 95.4% mean accuracy on permuted MNIST after 100 tasks, significantly surpassing individual methods.

Key insights

• Combining context-dependent gating and synaptic stabilization provides a more robust and scalable solution to catastrophic forgetting than either method used in isolation, reflecting biological learning principles.
• XdG creates sparse, task-specific neural activity patterns, effectively reserving a large proportion of synapses as a 'reservoir' for future tasks without interfering with prior knowledge.
• Networks employing XdG with SI exhibit lower mean synaptic importance values, allowing greater flexibility for large synaptic adjustments during new learning while protecting only the most critical, high-importance synapses.

Summary by alphaXiv.