Reproduce “Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siru Ouyang, Shuohang Wang, Minhao Jiang, Ming Zhong, Donghan Yu, Jiawei Han, Yelong Shen
Researchers from the University of Illinois Urbana-Champaign and Microsoft systematically investigated the impact of decoding temperature on speculative decoding performance when combined with knowledge distillation. Their findings reveal that aligning knowledge distillation training temperatures with inference decoding temperatures significantly enhances efficiency, leading to a proposed data composition strategy that improved speedup by 12-20% across various datasets.
Key insights
- Speculative decoding speedup consistently decreases as decoding temperature increases, largely due to heightened computational complexity at higher temperatures.
- Optimal speculative decoding performance is achieved when the knowledge distillation training temperature closely aligns with the inference decoding temperature, indicating the importance of matching probability distributions.
- Online distillation generally yields higher speedups than offline distillation, particularly at higher KD temperatures, suggesting benefits from softened probability distributions for learning teacher behavior.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.