Reproduce “ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wonjun Kang, Kevin Galim, Seunghyuk Oh, Minjae Lee, Yuchen Zeng, Shuibai Zhang, Coleman Hooper, Yuezhou Hu and 3 more
Researchers developed PARALLELBENCH, a novel benchmark, to systematically evaluate the speed-quality trade-offs of parallel decoding in Diffusion LLMs (dLLMs). They confirmed through information-theoretic analysis and empirical studies that dLLMs exhibit severe quality degradation on tasks with high token dependencies when generating in parallel, highlighting current decoding strategies' limitations.
Key insights
- Parallel decoding in dLLMs inherently introduces distribution error, which is lower-bounded by the sum of conditional total correlations and increases as the degree of parallelism intensifies.
- Current parallel decoding strategies and adaptive unmasking methods struggle to effectively balance generation speed and output quality, especially for tasks with high intrinsic token dependencies.
- Fine-tuning or training-free remasking samplers do not fundamentally overcome the quality degradation for highly dependent tasks under parallel decoding, supporting the theoretical limitations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.