Reproduce “Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junyi Wen, Junyuan Liang, Zicong Hong, Wuhui Chen, Ting Cai, Zibin Zheng
Researchers from Sun Yat-sen University and Peng Cheng Laboratory developed Krul, an LLM inference system that performs dynamic, conversation-adaptive KV cache compression to efficiently restore states in multi-turn conversations. This system reduces Time-to-First-Token by up to 2.68x, decreases KV cache storage by 1.33x to 2.35x, and improves prefill throughput by an average of 2.5x while maintaining generation quality with less than 1% accuracy loss.
Key insights
- Optimal KV cache compression is highly context-dependent, necessitating conversation-adaptive strategies to preserve accuracy, unlike fixed compression methods.
- LLM attention layers exhibit distinct processing behaviors, with 'Initial-Recent' (I-R) layers being largely invariant to future inputs, making their KV caches suitable for selective cross-layer sharing.
- Efficient dynamic compression and restoration for LLMs require a multi-faceted system design that combines intelligent compression strategy selection, heterogeneous computation, and optimized pipeline scheduling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.