Reproduce “Training Domain Draft Models for Speculative Decoding: Best Practices and Insights” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fenglu Hong, Ravi Raju, Jonathan Lingjie Li, Bo Li, Urmish Thakker, Avinash Ravichandran, Swayambhoo Jain, Changran Hu
Researchers at SambaNova Systems systematically investigated methods for training domain-specific draft models to enhance speculative decoding efficiency for Large Language Models. Their findings demonstrate that offline white-box distillation, utilizing the target model's internal logits, consistently yields higher token acceptance rates, improving performance by 2-25% over alternative methods. The study also shows that synthetic data can effectively align draft models, achieving 80-93% of the performance attained with in-domain user queries.
Key insights
- Offline distillation consistently outperforms online distillation, leveraging richer learning signals from static datasets to achieve higher acceptance rates.
- White-box distillation, especially using forward Kullback-Leibler (KL) divergence, provides a stronger supervisory signal and generally surpasses black-box SFT.
- Synthetic data generation techniques, like Magpie, are effective for aligning draft models in data-scarce scenarios, serving as a viable alternative to in-domain user queries.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.