Reproduce “Efficient Long Sequence Modeling via State Space Augmented Transformer” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, Jianfeng Gao

Researchers from Georgia Institute of Technology and Microsoft Research developed SPADE, a hybrid architecture that integrates state space models with efficient attention mechanisms to address the limitations of processing long sequences. This approach achieves state-of-the-art results on the Long Range Arena benchmark and demonstrates linear computational complexity, enabling efficient and accurate modeling of long-range dependencies.

Key insights

  • The State Space Model (SSM) component is most effective when placed at the bottom layer of the architecture, functioning as an initial global information extractor.
  • SPADE demonstrates robust and consistent performance as sequence length increases, unlike traditional Transformers which show performance degradation.
  • Local and global information processing are complementary; larger local attention window sizes can further improve performance when global context is provided by the SSM.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.