Reproduce “SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Divya Jyoti Bajpai, Kishan Kumar Upadhyay, Manjesh Kumar Hanawal
SPADE presents a distributed edge-cloud inference framework that optimizes Large Language Model (LLM) deployment by offloading speculative token generation to a lightweight edge model and using a large cloud model for verification. This approach achieves statistically identical output accuracy to a full cloud model while reducing cloud calls by 76-77% and increasing average throughput by over 30% across multiple NLP tasks.
Key insights
- Integrating speculative decoding into an edge-cloud distributed architecture effectively minimizes expensive cloud-based verifier calls while maintaining the output fidelity of the larger model.
- The framework operates in a 'plug-and-play' manner, allowing for immediate deployment with existing pre-trained LLMs without requiring additional training or fine-tuning.
- The number of speculative tokens generated per edge call (d) is a critical hyperparameter, directly influencing cloud model invocations and overall efficiency, with larger 'd' generally reducing cloud calls.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.