Reproduce “Self-Expanding Neural Networks” on OpenResearch

About this paper

Rupert Mitchell, Robin Menzenbach, Kristian Kersting, Martin Mundt

Self-Expanding Neural Networks (SENN) introduces a gradient-based framework for dynamically adjusting neural network architecture by growing and pruning neurons or layers during training. The method leverages a 'natural expansion score' to guide these modifications, enabling stable performance without disrupting ongoing learning and demonstrating improved accuracy across diverse tasks, including adapting pre-trained models.

Key insights

• The natural expansion score provides a unified metric that identifies when and where a network's current capacity is locally insufficient, guiding the optimal introduction of new parameters.
• By initializing new parameters strategically, network expansion (adding width or depth) can occur without disrupting the existing learned function or causing accuracy drops during training.
• Approximating second-order information (Fisher Matrix via KFAC) and deriving a computationally efficient lower bound for expansion score increase makes dynamic architecture optimization feasible for deep neural networks.

Summary by alphaXiv.