Reproduce “River-LLM: Large Language Model Seamless Exit Based on KV Share” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yingtao Shen, An Zou
River-LLM makes token-level early exit practical by using lightweight companion layers to supply approximate key and value entries for skipped decoder layers, sharing the backbone’s cache addressing. On GSM8K, Llama3.2 1B reached 182.9 tokens per second versus 84.5 for its backbone, a 2.16× speedup, with 29.3% versus 33.2% accuracy, at batch size 1 on an NVIDIA A40.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.