Reproduce “SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongliang Cao, Xiuyu Li, Siqi Wang, Ziming Wang, Jingmin Zhu, Hailong Yang
SpecFit selects a prefix-valid draft subtree for offloaded mixture-of-experts decoding, estimating expected output against predicted cost from selected nodes and distinct experts they call. It can stop drafting when another level does not improve this estimate, while the target model’s routing remains unchanged. With 48 expert slots per layer, it reached 8.27 decode tokens per second on CNN/DailyMail, 21.6% above cached EAGLE-3, despite fewer committed tokens per cycle.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.