Reproduce “NebulaSD: Many-for-Many Speculative Decoding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junhao He, Hongyang Du

NebulaSD separates draft and target workers into shared pools that independently reconstruct request batches, while preparing each request’s KV cache for reassignment and overlapping transfers with computation. On four RTX 4090 GPUs, it processes 1,914.83 speculative request rounds/s, versus 1,273.35 for Native Distributed. Broader proportional scaling appears only in simulation with idealized state movement.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.