Reproduce “Preserving Plasticity in Continual Learning via Dynamical Isometry” on OpenResearch

About this paper

Andries Rosseau, Robert Müller, Ann Nowé

Researchers established a theoretical framework connecting plasticity loss in continual learning to Neural Tangent Kernel anisotropy, using dynamical isometry as a preservation target. The proposed Gram deviation regularization and AdamO optimizer consistently outperformed existing methods and reduced inactive ReLU units across supervised and reinforcement learning benchmarks.

Key insights

• Plasticity loss in continual learning is fundamentally linked to the increasing anisotropy (high condition number) of the Neural Tangent Kernel, which hinders efficient gradient-based updates across output-space directions.
• Maintaining approximate dynamical isometry across network layers is a principled mechanism to ensure the NTK remains isotropic, thereby preserving the network's capacity for efficient learning.
• The proposed Gram deviation regularization directly contributes to reactivating dormant ReLU units by rotating inactive weight vectors, pushing them back into regions with positive pre-activations.

Summary by alphaXiv.