Reproduce “Softmax Linear Attention: Reclaiming Global Competition” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mingwei Xu, Xuan Lin, Xinnan Guo, Wanqing Xu, Wanyun Cui

Softmax Linear Attention (SLA), developed by researchers from Shanghai University of Finance and Economics and Ant Group, introduces head-level global competition to linear attention models. This approach enhances expressivity and robustness, leading to performance improvements in long-context retrieval, such as an average accuracy boost of +7.71% on real-world retrieval tasks, and perplexity reductions in language modeling benchmarks.

Key insights

  • Global competition can be effectively restored to linear attention models through head-level softmax, leveraging the multi-head architecture's inherent semantic specialization as a computationally efficient alternative to token-wise competition.
  • The head-level softmax gates restore magnitude sensitivity, allowing the model to dynamically sharpen its focus based on query/key confidence, and enable an asymptotic 'winner-take-all' dynamic for information flow.
  • The multi-head architecture provides a natural and cost-effective granularity for introducing competitive selection, ensuring that information is routed only if reading and writing heads align on the same semantic subspace.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.