Reproduce “CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotion” on OpenResearch

About this paper

Oswaldo Ludwig

CLASSP, a training method inspired by biological synaptic plasticity, mitigates catastrophic forgetting in neural networks through mechanisms for adjustment suppression and sparsity promotion. It achieved higher accuracy on previously learned tasks and demonstrated superior memory efficiency compared to existing continual learning approaches in both computer vision and text classification benchmarks.

Key insights

• Biological principles from synaptic plasticity, such as decay rates and threshold effects, can be effectively translated into algorithmic mechanisms to combat catastrophic forgetting.
• Generalizing the AdaGrad optimizer with a custom power parameter `p=1` and a gradient threshold provides a more effective and balanced approach for adaptive learning rates in continual learning scenarios.
• CLASSP offers a memory-efficient solution for continual learning, requiring only one scaling factor per weight, in contrast to methods like EWC which demand more extensive memory for past model parameters or matrices.

Summary by alphaXiv.