Reproduce “Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks” on OpenResearch

About this paper

Julia Gygax, Friedemann Zenke

The paper by Gygax and Zenke at Friedrich Miescher Institute for Biomedical Research provides a theoretical foundation for surrogate gradient (SG) learning in spiking neural networks (SNNs), demonstrating its derivation from the Stochastic Automatic Differentiation (stochAD) framework for stochastic SNNs. This work empirically validates the effectiveness of SGs in training stochastic SNNs on tasks like spike train matching and SHD classification, achieving competitive performance (e.g., 97.2% validation accuracy on SHD) while preserving network stochasticity.

Key insights

• Surrogate gradients in stochastic SNNs are theoretically equivalent to smoothed stochastic derivatives derived from the stochAD framework, offering a principled justification.
• Deterministic SNNs can be conceptualized as a limiting case of stochastic SNNs, with their SGs operating as point estimates of these stochastic gradients.
• Surrogate gradients generally do not form conservative vector fields and are not gradients of an explicit scalar surrogate loss function, often failing to preserve the sign of true gradient components.

Summary by alphaXiv.