Reproduce “Jet-QSparse: Accelerating Agentic LLM Inference with 4-Bit Sparse Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haozhan Tang, Hao Kang, Meng Lu, Yecheng Wu, Zerui Wang, Seokho Han, Haocheng Xi, Shuo Yang and 2 more
Jet-QSparse accelerates decode for long-context attention while retaining every cached token. It stores keys and values in NVFP4, scans keys to rank blocks, then performs higher-precision value aggregation on selected blocks. At 128K context and batch 16, it achieves 1.94× decode TPOT speedup over dense BF16 with 64 blocks per KV head; this does not establish faster prefill or an accuracy gain.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.