Reproduce “Self-Speculative Masked Diffusions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andrew Campbell, Valentin De Bortoli, Jiaxin Shi, Arnaud Doucet
The authors present Self-Speculative Masked Diffusions: a masked-diffusion network drafts tokens, and a small causal component checks them in generation order. On OpenWebText, it reaches GPT-2’s NLL of 5.12 nats per dimension at 64 NFE, matching the baseline’s 5.13 at 128. For proteins, the method reports about a 2× NFE reduction at high pLDDT, a predicted-confidence proxy.
Key insights
- A single hybrid Transformer architecture can effectively parameterize both a fast, factorized 'draft' distribution and a more accurate, non-factorized 'target' distribution.
- Adapting speculative sampling to MDMs allows for efficient access to the non-factorized target distribution, enabling parallel verification of drafted tokens and significant NFE reduction.
- A residual connection between non-causal and causal layers is critical for training the causal distribution and enhancing the acceptance rate during speculative sampling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.