Reproduce “Beyond the Target: From Imitation to Collaboration in Speculative Decoding” on OpenResearch

About this paper

Jinze Li, Yixing Xu, Guanchen Li, Jinfeng Xu, Shuo Yang, Yang Zhang, Xuanwu Yin, Dong Li and 2 more

CoSpec introduces a collaborative speculative decoding framework for Large Language Models that, unlike traditional methods, leverages disagreements between a draft and target model to enhance final task utility while accelerating inference. This approach achieves up to 4x speedup and improves target model performance by 1.7% to 2.9% on reasoning and code generation benchmarks.

Key insights

• Draft models, despite being generally weaker, are not uniformly inferior at every token-level decision and can propose alternative tokens that lead to more correct or useful final answers.
• Disagreements between draft and target models should be re-framed as opportunities for collaboration to improve final task utility, rather than simply as verification failures to be corrected by the target.
• A learned arbitration mechanism can effectively leverage the non-nested strengths of different models at the token level during decoding, enabling the system to surpass the performance ceiling of the target model alone.

Summary by alphaXiv.