Reproduce “Rethinking Soft Tokens for Parallel Decoding in Diffusion Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kodai Kawamura, Kenji Kawaguchi, Anji Liu
The method improves parallel decoding in frozen pretrained language models by representing unresolved tokens with prediction-informed embeddings. It averages candidate embeddings, then uses spherical interpolation toward their aggregate while preserving the [MASK] embedding’s norm. Across benchmarks, it outperforms two baselines in most settings; on Dream’s HumanEval test, it reaches 32.32% pass@1, versus 17.68% and 23.78%.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.