Reproduce “Pretraining Without Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junxiong Wang, Jing Nathan Yan, Albert Gu, Alexander M. Rush
Researchers at Cornell University and DeepMind developed Bidirectional Gated SSM (BiGS), an architecture that achieves performance comparable to BERT on standard NLP benchmarks without self-attention, utilizing State-Space Models combined with multiplicative gating. The model demonstrated effective processing of sequences up to 4096 tokens and exhibited distinct inductive biases for syntactic understanding.
Key insights
- Multiplicative gating is essential for SSM-based models to achieve high transferability and generalize effectively to downstream tasks, beyond simply reducing pretraining loss.
- SSM-based architectures possess distinct inductive biases, demonstrating a superior capacity for processing certain syntactic structures, such as subject-verb agreement with attractors.
- The inherent linear complexity of SSMs enables efficient scaling to extremely long sequence lengths (up to 4096 tokens) without requiring approximation methods common in attention-based models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.