Reproduce “Dynamically Modular and Sparse General Continual Learning” on OpenResearch

About this paper

Arnav Varma, Elahe Arani, Bahram Zonooz

Dynamos, a method from NavInfo Europe and Eindhoven University of Technology, integrates rehearsal-based learning with dynamic modularity and sparsity, inspired by biological brains. This approach enables deep neural networks to learn continuously from sequential data, reducing catastrophic forgetting by adaptively activating relevant subsets of neurons. Dynamos achieved up to 84.815% accuracy on Seq-SVHN and 90.519% on MNIST-360, outperforming established baselines in general continual learning settings.

Key insights

• Incorporating biologically inspired dynamic, sparse, and modular processing can effectively mitigate catastrophic forgetting in continual learning by selectively activating relevant network components.
• Combining a policy-gradient trained channel gating mechanism with rehearsal-based learning allows for efficient reuse and specialization of network modules without requiring unbounded network growth or explicit task information.
• Encouraging prototype clustering of channel-wise attention vectors based on semantic similarity helps the network learn input-adaptive modules that organize information effectively across classes and tasks.

Summary by alphaXiv.