Reproduce “Understanding the Convergence in Balanced Resonate-and-Fire Neurons” on OpenResearch

About this paper

Saya Higuchi, Sander M. Bohte, Sebastian Otte

This research investigates why Balanced Resonate-and-Fire (BRF) neurons exhibit faster and more stable convergence in recurrent Spiking Neural Networks (SNNs). It demonstrates that BRF networks possess a smooth error landscape and maintain stable gradient flow, achieving superior training speed and stability compared to Adaptive Leaky Integrate-and-Fire (ALIF) and even some Long Short-Term Memory (LSTM) models.

Key insights

• BRF networks exhibit a smooth, flat, and nearly convex error landscape with a wide basin, which facilitates easier and more stable optimization compared to the rougher, chaotic landscapes of ALIF and vanilla RF neurons.
• The divergence boundary component of the BRF neuron has a predominant impact on convergence speed and stability by directly regulating the spectral radius of the internal state transition matrix.
• Formal analysis confirms that BRF neurons inherently support stable gradient flow by ensuring the spectral radius of their internal state transition matrix remains at or below unity, preventing vanishing or exploding gradients during BPTT.

Summary by alphaXiv.