Reproduce “When Drafts Evolve: Speculative Decoding Meets Online Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yu-Yang Qian, Hao-Cong Wu, Yichao Fu, Hao Zhang, Peng Zhao
OnlineSPEC recasts speculative decoding as an online learning problem in which target-model verification continuously updates the draft model. Its gradient, optimistic, and ensemble variants improve accepted sequence length and inference speed across reasoning, coding, text-to-SQL, and finance tasks, with reported gains including up to 1.90× speedup for Opt-Hydra and 1.41× for Online-LR, while largely preserving task accuracy.
Key insights
- Dynamic rather than static regret is the appropriate comparison because the best draft parameters can vary with incoming prompts and the limited draft architecture may only approximate the target locally.
- The useful optimization target is system-level acceptance and throughput, not draft loss alone; lower draft-target divergence increases accepted tokens per verification round and potential speedup.
- Architecture-specific feedback matters: semantic preference signals benefit reasoning-oriented drafts, predictive gradients help Hydra, and online ensembles reduce sensitivity to a single learning rate.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.