Reproduce “BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hayden Prairie, Jacob M. Ryan, Taylor Berg-Kirkpatrick, Daniel Y. Fu
The authors introduce BeNice!, a fine-tuning method that trains looped language models to use the final recurrence’s key–value cache in earlier recurrences, reducing cache storage without removing recurrent computation. On a 770M model, shared-cache validation perplexity is 11.86 versus 21.61 after standard finetuning. Simulated serving shows 23.6% higher throughput at 2 requests per second; at low load, performance matches baseline.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.