Reproduce “Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Fang Wan, Xufeng Liu, Fan Li, Yi Liu

MASA changes sparse-attention ranking to favor units that better preserve the matrix-product output over those with the largest attention values. It keeps existing sparse units, kernels, and budgets. It improves all eight matched method–benchmark average scores. A 64K RULER diagnostic finds a 7.17% reduction in normalized attention-output error for FlexPrefill with Qwen2-7B-Instruct. These findings concern sparse prefill and a diagnostic setting.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.