Reproduce “Twindex: Index Cache Offloading with Coarse-to-Fine Selection for Long-Context Sparse-Attention Serving” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuxuan Xia, Jiakun Fan, Han Meng, Hao Mark Chen, Shenji Wan, Dongsheng Yang

Twindex reduces GPU-memory use for sparse-attention decoding by screening history entries with a compact index copy, then rescoring candidates against native index rows before final selection. It stores compressed keys in GPU memory and native rows in host memory, with a hot buffer for rows. In a 992K-token Qwen run, throughput rose 1.438×; results are limited to tested serving setups.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.