Reproduce “Hungry Hungry Hippos: Towards Language Modeling with State Space Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, Christopher Ré

Researchers from Stanford University and University at Buffalo introduce H3, a novel State Space Model (SSM) layer, and FlashConv, an efficient training algorithm, to enhance SSMs for large-scale language modeling. The combined approach enabled SSMs to match or surpass Transformer performance, achieving lower perplexity and up to 2.4x faster text generation on various benchmarks, while also demonstrating strong performance on diverse non-text modalities.

Key insights

  • Existing State Space Models (SSMs) underperform attention in language modeling due to a lack of specific expressivity for associative recall and token comparison across sequences.
  • Carefully designed SSM architectures, such as H3, can incorporate the necessary capabilities to match attention's performance on key language modeling tasks.
  • Hardware-aware algorithm design (FlashConv) is crucial for SSMs to achieve practical efficiency, enabling linear scaling and high GPU utilization that surpasses even optimized Transformer implementations for long sequences.
  • SSMs and attention exhibit complementary strengths, allowing hybrid architectures to outperform pure Transformer models in language modeling performance and efficiency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.