Reproduce “Human-Level Control through Directly-Trained Deep Spiking Q-Networks” on OpenResearch

About this paper

Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, Huajin Tang

This research from Southwestern University of Finance and Economics, University of Electronic Science and Technology of China, and Zhejiang University presents a directly-trained Deep Spiking Q-Network (DSQN) that achieves human-level control on Atari games. The DSQN operates with a significantly shorter simulation time window of 64 timesteps, demonstrating competitive performance compared to conventional Deep Q-Networks and superior performance over existing conversion-based spiking neural network methods while being more energy efficient.

Key insights

• Direct training of Spiking Neural Networks (SNNs) for complex reinforcement learning (RL) tasks like Atari games is viable and effective, moving beyond the limitations of ANN-to-SNN conversion methods.
• Leaky Integrate-and-Fire (LIF) neurons are more suitable for directly-trained SNNs in Deep Reinforcement Learning (DRL) due to their flexibility and ability to operate effectively with short simulation windows without strict input normalization.
• Human-level control in DRL can be achieved with significantly reduced computational cost and energy consumption by leveraging directly-trained SNNs operating with very short simulation time windows.

Summary by alphaXiv.