Reproduce “ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qiuyang Zhang, Kai Zhou, Ding Tang, Kai Lu, Cheng Li, Zhenyu Yang, Peng Xu, Jiguang Wan
ScoutAttention splits block-wise sparse attention between CPU and GPU, keeping selected key/value blocks and digests on GPU while the CPU computes selected offloaded blocks. It starts CPU work one layer ahead and periodically recalls blocks. With Qwen 3 14B, it achieves 5.1× FullKV decode throughput at 64k input length; separate Qwen 3 8B LongBench testing reports a 2.1% average accuracy decrease versus full attention at a 2,048-token budget.
Key insights
- CPU attention computation can have higher throughput than PCIe data transfer, favoring co-attention strategies over recall-based methods for offloading.
- Predicted queries, derived from the current layer's input and the next layer's projection matrix, exhibit high cosine similarity to actual queries, enabling effective layer-ahead pre-computation.
- Managing the dynamic nature of important KV blocks through asynchronous periodic recall is crucial for maintaining CPU efficiency and balancing the computational load without frequent GPU stalls.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.