Reproduce “PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding” on OpenResearch

About this paper

Qiuyang Zhang, Kai Zhou, Kai Lu, Haocheng Lu, Jian Zhou, Yuanpeng Su, Kun Bao, Jiguang Wan and 1 more

PulseInfer treats CPU–GPU KV-cache recall as a scheduling and data-transfer problem, combining asynchronous layer-wise execution, adaptive request admission, and coalesced block retrieval for long-context LLM decoding. Across evaluated models and workloads, it delivers 1.5×–4.7× the decode throughput of standard SGLang and 1.1×–2.6× that of the strongest tested offloading baseline, while retaining higher per-token latency than full-cache inference on average.

Key insights

• The cost of sparse KV offloading depends not only on how many tokens are attended to, but also on the size, timing, and overlap of CPU–GPU transfers.
• Separating normal and offload batches can reduce short requests’ exposure to recall stalls while keeping GPU computation active during transfers.
• Throughput gains come with a tradeoff: PulseInfer improves aggregate serving capacity under memory pressure but does not match full-cache inference’s average per-token latency.

Summary by alphaXiv.