Reproduce “CISA: Retrofitting Dense Attention Models with Chunk-Indexed Sparse Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohammed Dabbah, Izhak Golan, Oren Tropp, Daniel Afrimi, Vladimir Anisimov, Tomer Asida, Nave Assaf, Ran El-Yaniv and 5 more
CISA reduces long-context language model cache reads by selecting relevant 16-token chunks, while retaining the full KV history. It chooses chunk selectors from calibration activations without gradient updates, and shares selections across adjacent prefill queries. On Ultra, QShare-4 achieves a 47.68% four-benchmark mean versus 48.04% for dense attention, with 2.285× fixed-output throughput; longer generations lower the adjusted proxy to 1.939×.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.