Reproduce “More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiebin Zhang, Dawei Zhu, Yifan Song, Wenhao Wu, Chuqiao Kuang, Xiaoguang Li, Lifeng Shang, Qun Liu and 1 more
A novel 'quantized pruning' strategy for large language model KV caches demonstrates that storing more tokens at lower precision consistently outperforms retaining fewer tokens at full precision under the same memory budget. This approach improves long-context performance across various tasks, input lengths, and model scales, particularly for retrieval-heavy applications.
Key insights
- Integrating KV pruning and KV quantization is feasible, achieving lower overall compression rates with minimal performance degradation when precision is reduced to 8-bit or 4-bit.
- Under equivalent memory budgets, storing more tokens at lower precision ("quantized pruning") consistently outperforms retaining fewer tokens at full 16-bit precision across various tasks and models.
- Reducing the number of preserved tokens results in a more significant performance drop than reducing precision (to a certain extent), indicating that token coverage often plays a more critical role than precision under memory constraints.
- Intermediate layers within LLMs show greater sensitivity to token retention, benefiting more significantly from preserving a larger number of tokens at lower precision compared to initial or final layers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.