Reproduce “Fluctuation-driven initialization for spiking neural network training” on OpenResearch

About this paper

Julian Rossbroich, Julia Gygax, Friedemann Zenke

A fluctuation-driven initialization strategy for Spiking Neural Networks (SNNs), based on analytical derivations for membrane potential dynamics, provides a robust approach for training. This method, introduced by researchers at Friedrich Miescher Institute for Biomedical Research, consistently improves learning performance in diverse deep SNN architectures and datasets, effectively mitigating vanishing gradient problems compared to conventional initializations.

Key insights

• Deep Spiking Neural Networks are highly sensitive to initial weight distributions, with training success dependent on initializing neurons in a specific fluctuation-driven regime.
• Training failures in deep SNNs with suboptimal initialization are primarily caused by vanishing surrogate gradients due to quiescent hidden layers or exploding gradients from saturated firing rates.
• Incorporating homeostatic plasticity or using specific optimizers can increase the robustness of SNN training to initialization, rescuing performance in suboptimally set networks.

Summary by alphaXiv.