Reproduce “Steering Pretrained Drafters during Speculative Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Frédéric Berdoz, Peer Rheinboldt, Roger Wattenhofer
Researchers from ETH Zurich present SD 2, a method to dynamically align pretrained drafters with a verifier in speculative decoding by injecting a steering vector derived from the verifier's hidden states. This approach increased the number of accepted tokens per block by 21% on average compared to distilled drafters, leading to an average speedup of 16-20% across various LLM configurations and tasks.
Key insights
- Large language models implicitly encode predictive information about upcoming tokens within their intermediate representations, which can be extracted.
- Dynamically steering a pretrained drafter with signals from the verifier's internal state can effectively combine the generalization benefits of strong pretrained models with real-time alignment.
- A simple conditional bias injected into the drafter's MLP layers is sufficient to achieve significant alignment improvements with negligible computational overhead.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.