Reproduce “LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Renyuan Liu, Yuyang Leng, Kaiyan Liu, Yuzhou Zhong, Shaohan Hu, Chun-Fu, Chen, Peijun Zhao and 2 more
LeanStream coordinates on-device language model inference by predicting important weights early, then revising priorities as partial GPU results arrive. It loads and computes ready weights while others are still moving from storage. On Jetson AGX Orin, Mistral-7B on Scrolls-Qasper achieves 1.6× throughput at the same 3 GB budget, and a 5.8× lower budget at matched throughput. Results vary across tested settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.