Reproduce “Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shikuang Deng, Yuhang Li, Shanghang Zhang, Shi Gu

Researchers from the University of Electronic Science and Technology of China, Yale University, and Peking University developed Temporal Efficient Training (TET) and Time Inheritance Training (TIT) algorithms to enhance the direct training of Spiking Neural Networks (SNNs). These methods improve generalization by guiding SNNs to flatter loss landscape minima and significantly reduce training time, achieving state-of-the-art accuracy on datasets like CIFAR10 (94.50% with T=6) and DVS-CIFAR10 (83.17%), substantially closing the performance gap with Artificial Neural Networks.

Key insights

  • Standard direct training with surrogate gradients leads to poor generalization because the averaged error term quickly diminishes, trapping the optimizer in sharp local minima.
  • Calculating and averaging losses at each individual time step (TET) maintains stronger, more consistent gradient signals, effectively guiding the network to converge into flatter, more generalizable regions of the loss landscape.
  • TET inherently improves the temporal scalability of SNNs, allowing models trained at shorter simulation lengths to perform robustly when simulation length is increased without extensive finetuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.