Reproduce “HiState: Hierarchical Prefix Cache Management for Hybrid LLM Serving in Agentic Workloads” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zaifeng Pan, Lucas Wang, Xinwei Qiang, Yanbo Zhou, Chang Chen, Zhengding Hu, Yue Guan, Yufei Ding
The authors introduce HiState, a cache system for hybrid language models that combines attention caches with recurrent checkpoints. It keeps key–value data in faster memory and checkpoints on SSD, using adaptive placement and eviction to balance reuse and storage. On coding-agent traces using Qwen3.5-9B on one H100, it achieves up to 5.73× and 5.44× p99 TTFT speedups over vLLM-Marconi and SGLang-HiCache.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.