Reproduce “Feedforward Sequential Memory Neural Networks without Recurrent Feedback” on OpenResearch

About this paper

ShiLiang Zhang, Hui Jiang, Si Wei, LiRong Dai

Feedforward Sequential Memory Networks (FSMNs) capture long-term dependencies in sequential data by integrating learnable memory blocks within a feedforward architecture. This approach avoids the complexities of recurrent neural networks while achieving superior performance, such as 102 perplexity on the Penn Treebank and 92 perplexity on the enwik9 language modeling benchmarks.

Key insights

• Feedforward Sequential Memory Networks (FSMNs) function as Finite Impulse Response (FIR) filters, providing inherent stability and avoiding the instability issues of Infinite Impulse Response (IIR) filter-like RNNs.
• Long-term dependencies can be effectively learned by feedforward networks through learnable memory blocks, challenging the necessity of recurrent feedback for state-of-the-art sequential modeling.
• The memory blocks can be trained using standard back-propagation, which eliminates the complexities of Back-Propagation Through Time (BPTT) and enables better parallelization during training.

Summary by alphaXiv.