Reproduce “SMTrack: End-to-End Trained Spiking Neural Networks for Multi-Object Tracking in RGB Videos” on OpenResearch

About this paper

Pengzhi Zhong, Xinzhe Wang, Dan Zeng, Qihua Zhou, Feixiang He, Shuiwang Li

SMTrack presents the first directly-trained deep spiking neural network (SNN) framework for end-to-end multi-object tracking (MOT) in standard RGB videos. It achieves performance competitive with state-of-the-art artificial neural network (ANN)-based methods, often surpassing them in Higher Order Tracking Accuracy (HOTA) and identity metrics across various benchmarks, while operating with low temporal timesteps suitable for energy-efficient deployment.

Key insights

• Directly trained Spiking Neural Networks (SNNs) can perform complex temporal Multi-Object Tracking (MOT) tasks on RGB video, achieving performance comparable to state-of-the-art ANN-based methods with minimal temporal timesteps (e.g., T=2 or T=4).
• Dynamically adjusting the normalization factor in loss functions, as with the proposed Asa-NWDLoss, significantly improves detection and localization accuracy by adapting to batch-specific object scale distributions rather than relying on fixed global constants.
• A hybrid approach, combining an energy-efficient SNN-based detector with robust, training-free identity association modules (like TrackTrack), offers a pragmatic pathway to deploy high-performance, low-power MOT systems.

Summary by alphaXiv.