Reproduce “Millrace: Efficient Pipelined Training for Speculative Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Si Shen, Chen Chen, Hao-Yuan He, Jiajie Huang, Pengfei Liu, Tingxuan Han, Jiaqi Zheng, Runzhou Liu and 4 more
Millrace coordinates feature preparation and draft-model training for speculative decoding. It replays the training sampler to anticipate upcoming records, retains reusable target-model features in shared HBM, and schedules dependent operations around complete training inputs. In a Qwen3-8B/DSpark evaluation at 16K sequence capacity, it reduced end-to-end completion time by 57.6% versus vLLM-Spec; this is not a training-only speedup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.