Reproduce “Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment” on OpenResearch

About this paper

Chengting Yu, Xiaochen Zhao, Lei Liu, Shu Yang, Gaoang Wang, Erping Li, Aili Wang

Researchers at Zhejiang University developed a temporal-wise distillation framework for deep Spiking Neural Networks (SNNs), which leverages spatio-temporal dynamics to enhance classification accuracy and enable flexible operation across a full range of inference timesteps without requiring retraining. The method achieved 97.00% accuracy on CIFAR-10 (ResNet-19, T=6) and 71.04% on ImageNet (ResNet-34, T=4), demonstrating stable performance across varying inference lengths with negligible additional training cost.

Key insights

• Explicitly leveraging the spatio-temporal dynamics of SNNs by optimizing loss objectives at each individual timestep significantly improves performance and robustness.
• Self-distillation from the SNN's own ensemble output serves as an effective regularization mechanism, aligning individual timestep performances with the more robust aggregated output.
• The temporal-wise distillation approach ensures that a single SNN model, trained at a maximum timestep, can maintain high accuracy across a full range of shorter inference timesteps without specific retraining.

Summary by alphaXiv.