Reproduce “Twindex: Index Cache Offloading with Coarse-to-Fine Selection for Long-Context Sparse-Attention Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxuan Xia, Jiakun Fan, Han Meng, Hao Mark Chen, Shenji Wan, Dongsheng Yang
Twindex reduces GPU-memory use for sparse-attention decoding by screening history entries with a compact index copy, then rescoring candidates against native index rows before final selection. It stores compressed keys in GPU memory and native rows in host memory, with a hot buffer for rows. In a 992K-token Qwen run, throughput rose 1.438×; results are limited to tested serving setups.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.