Reproduce “BasisKV: Value Latents for Communication and Key Routing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lixun Zhang, Zishan Shao, Kangning Cui, Yu Liang, Fan Yang, Yiran Chen
BasisKV jointly fits a low-rank Value representation to reduce both KV-cache storage and tensor-parallel communication, and reuses it to help route Key pages. It preserves GPU contributions separately during communication and reconstructs the output afterward. On Llama-3.1-8B-Instruct, V64 achieved 1.42× end-to-end speedup at batch size 256 under the reported eight-L40S setup; gains vary by batch and Value width.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.