Reproduce “ResMISA: Residual Mixture of Sparse Attention for Long-Context LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruijie Zhou, Fanxu Meng, Zhixin Pan, Jinbao Xue, Ke Zhang, Jun Yu, Wenjie Pei
ResMISA reduces token-scoring channels by combining every head’s linear contribution into one shared score and routing nonlinear corrections. On DeepSeek-V3.2, RULER scores are 95.56 for ResMISA and 95.62 for DSA through 128K. At 1M, ResMISA with LiteTopK measures 558.18 seconds to first token versus DSA’s 1,358.91; this is not a quality result, and downstream gains after indexer training remain unestablished.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.