Reproduce “Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xingyu Xie, Zhaochen Yu, Yue Liao, Tao Wang, Kim-Chuan Toh, Shuicheng Yan
The Slow-Fast Inference (SFI) framework, developed by researchers from the National University of Singapore and ByteDance, accelerates large language model autoregressive decoding by decoupling generation into frequent low-cost sparse "fast steps" and occasional full-context "slow steps". This training-free method achieves throughput gains of 1.6x to 14.4x across various models and context lengths, while consistently maintaining or slightly improving task quality on long-context benchmarks.
Key insights
- LLM attention exhibits "within-sentence support stability," meaning the dominant attention mass often remains largely stable across consecutive decoding steps within semantically coherent spans.
- This temporal stability allows for an event-driven decoding strategy that avoids redundant full-history attention computations during periods of stable attention focus.
- A principled, closed-form Selector combining attention evidence with cache-aware priors and score refinements can effectively distill pertinent long-range tokens into a compact, sparse memory.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.