Reproduce “KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing” on OpenResearch

About this paper

Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, Zhi Chen

KVSharer, developed by researchers from Shanghai Jiao Tong University and ByteDance among others, introduces a training-free, layer-wise KV cache sharing method for large language models. The approach leverages a counterintuitive strategy of sharing dissimilar KV caches, achieving up to a 36% reduction in total memory and a 1.6x generation speedup for Llama2-13B-Chat while preserving model performance.

Key insights

• Sharing *dissimilar* KV caches across layers is more effective at preserving LLM performance than sharing similar ones, challenging conventional wisdom in model compression and sharing.
• An iterative, dissimilarity-based search strategy, guided by a final hidden-state similarity threshold, can identify effective layer-wise KV cache sharing patterns without needing costly model retraining.
• The method demonstrates robustness across various LLM architectures and sizes, and is orthogonal to existing intra-layer KV cache compression techniques, allowing for combined optimization.

Summary by alphaXiv.