Reproduce “Lost in a Few Heads: Diagnosing and Repairing Block Selection in Sparse Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenhe Zhang, Boxing Chen
The authors diagnose why deployed block-sparse attention misses relevant context blocks and test training-free repairs. They identify heads whose attention favors a needed block even when the selector omits it, linking many misses to pooled-key dilution. On MiniCPM4.1-8B, a 4-bit scorer raises 64K MK3 accuracy from 54.1% to 97.5%; this result covers tested checkpoints and tasks, not all sparse selectors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.