Reproduce “SPIRe: Boosting LLM Inference Throughput with Speculative Decoding” on OpenResearch

About this paper

Sanjit Neelam, Daniel Heinlein, Vaclav Cvicek, Akshay Mishra, Reiner Pope

SPIRe, developed by MatX, is a novel speculative decoding approach that significantly boosts Large Language Model inference throughput by optimizing draft model architecture for large batch sizes and varying context lengths. It achieves a 2.78x speedup for typical scenarios (B=64, L=512), representing a 100% increase over vanilla speculative decoding and a 35.6% increase over MagicDec, primarily by reducing the cost of draft model operations through a sparse KV cache and effective knowledge distillation from the target model.

Key insights

• A significantly lower iteration time multiplier ("t) for the draft model can outweigh a slightly lower average generation length (t), leading to superior overall throughput.
• The sparse KV cache (e.g., StreamingLLM attention) is the most critical component for boosting throughput in large-batch, long-context speculative decoding, enabling performance to scale with batch size.
• Distilling knowledge from the target model's internal activations during draft model training, rather than just its output logits, substantially improves the draft model's quality and acceptance rates

Summary by alphaXiv.