Reproduce “NebulaSD: Many-for-Many Speculative Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junhao He, Hongyang Du
NebulaSD separates draft and target workers into shared pools that independently reconstruct request batches, while preparing each request’s KV cache for reassignment and overlapping transfers with computation. On four RTX 4090 GPUs, it processes 1,914.83 speculative request rounds/s, versus 1,273.35 for Native Distributed. Broader proportional scaling appears only in simulation with idealized state movement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.