Reproduce “Fast State Restoration in LLM Serving with HCache” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiwei Gao, Youmin Chen, Jiwu Shu
HCache is an approach from Tsinghua University for rapid state restoration in Large Language Model (LLM) serving that utilizes intermediate hidden states to efficiently regenerate context. This method substantially reduces Time to First Token (TTFT) by up to 5.73x compared to recomputation and 1.93x compared to KV offload, while consuming less storage.
Key insights
- Intermediate hidden states are significantly more compact than the full KV cache, making them efficient to store and transfer for restoration.
- Hidden states can be leveraged as starting points to regenerate the KV cache with minimal computational effort, balancing I/O and compute.
- Dynamically profiling hardware and optimizing state restoration partitioning can reduce pipeline bubbles and maximize resource utilization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.