Reproduce “PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ao Wang, Hui Chen, Jiaxin Li, Jianchao Tan, Kefeng Zhang, Xunliang Cai, Zijia Lin, Jungong Han and 1 more
PrefixKV introduces an adaptive layer-wise KV cache retention strategy to optimize the inference efficiency of large vision-language models, preserving generation quality while significantly reducing memory overhead and improving throughput by up to 1.8x. The method dynamically allocates cache sizes to individual Transformer layers based on their unique importance distributions, achieving state-of-the-art compression performance.
Key insights
- KV vectors exhibit notably distinct importance distributions across different Transformer layers, making uniform compression strategies suboptimal.
- The cumulative priority sequences of KV vectors per layer are robust and similar across diverse input samples, enabling efficient offline estimation of optimal layer-wise cache sizes.
- By adaptively configuring KV cache size for each layer, it is possible to maintain high generation quality while achieving aggressive compression ratios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.