Reproduce “Probing the Limits of Compressive Memory: A Study of Infini-Attention in Small-Scale Pretraining” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ruizhe Huang, Kexuan Zhang, Yihao Fang, Baifeng Yu

Researchers at Huawei Technologies Canada explored the integration of Infini-attention into a 300-million-parameter LLaMA model to improve long-context understanding in Small Language Models (SLMs). This architectural modification enhanced training stability and, after fine-tuning, led to an increase of up to 31% in long-context retrieval accuracy compared to the baseline model.

Key insights

  • Infini-attention implicitly regularizes gradient flow, leading to improved training stability and enabling stable training at default learning rates for SLMs.
  • The trainable balance factor converged to favor local attention (70%), with memory utilization concentrated in early layers, suggesting hierarchical processing despite short-sequence pre-training data.
  • Architectural innovations designed for large-scale models can effectively transfer to and enhance the capabilities of small-scale language models, demonstrating architectural scalability beyond sheer model size.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.