Reproduce “Switching Linear Attention” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet, E. Kelly Buchanan, Emily B. Fox, Scott W. Linderman

SwiLA is a recurrent sequence layer that retrieves context-dependent key–value associations. It maintains several linear regressors and uses prediction errors and learned routing priors to update and combine them, keeping state size fixed. On a synthetic four-context task, one-head, four-mixture Temporal SwiLA reaches near-perfect accuracy at state size 2¹². In language-model recall tests, it scores 20.6, below FoX’s 33.7.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.