Reproduce “SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks” on OpenResearch

About this paper

Xiubo Liang, Jinxing Han, Yuke Li, Haoqi Zhu, Yu Zhao, Hongzhi Wang

The SMM Transformer introduces a Spiking Neural Network (SNN)-based architecture for multimodal tasks, leveraging learnable spiking neurons, sparse token mixing, and modality-aware fusion to reduce computational energy. It achieves accuracy comparable to traditional Artificial Neural Networks (ANNs) while cutting the attention block's arithmetic energy by up to 97% and overall model energy by over 20% on various visual and vision-language benchmarks.

Key insights

• Learnable parameters within parallel spiking neurons (PLMP) enable stable training for deep SNNs and allow for modeling heterogeneous temporal dynamics, reducing reliance on manual hyperparameter tuning.
• Sparse, spike-driven channel-wise co-activation (SMSA) can effectively replace dense Transformer attention, yielding substantial energy savings while maintaining expressive token mixing through a self-compensation branch.
• Modality-aware expert routing (SMoE) improves multimodal fusion in SNNs by dedicating specialized processing capacity to different input types, thus mitigating cross-modal interference.

Summary by alphaXiv.