Reproduce “Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer
Muir pairs matched-filter query readout with lower-rank erase and decay gates to reduce interference in a fixed-size linear-attention state, reallocating parameters to a wider feed-forward network. After 100B-token pretraining, its 1.34B-parameter model achieves lower FineWeb-Edu validation NLL than GDN-2. On a five-hop variable-tracking test at 4K context, it scores 24.0% versus 10.5% for GDN-2 and 21.5% for Transformer; this is not evidence of general multi-hop reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.