Reproduce “P/D-Device: Disaggregated Large Language Model between Cloud and Devices” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yibo Jin, Yixu Xu, Yue Chen, Chengbin Wang, Tao Wang, Jiaqi Huang, Rongfei Zhang, Yiming Dong and 22 more
Huawei Technologies developed P/D-Device, a cloud-device disaggregated framework for Large Language Models that offloads initial prefill and token generation to the cloud while devices handle subsequent decoding. This approach reduces user-perceived Time To First Token (TTFT) by at least 60% on devices, increases cloud throughput by up to 15x, and improves on-device inference quality, all while maintaining KB-sized data transfers.
Key insights
- Disaggregating LLM inference between cloud and device by having the cloud assist the device's prefill phase significantly improves user experience and cloud resource utilization simultaneously.
- Intelligent prompt refinement using attention scores allows for drastic reduction of on-device prefill computation by processing a shorter, critical prompt portion.
- Amortizing the on-device prefill latency with cloud-generated initial tokens can effectively hide the device's slower processing, delivering a perceptually fast response to the user.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.