Reproduce “Multi-Block Diffusion Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yijie Jin, Jiajun Xu, Yuxuan Liu, Chenkai Xu, Yi Tu, Jiajun Li, Dandan Tu, Xiaohui Yan and 3 more

Researchers from Shanghai Jiao Tong University and Huawei developed Multi-Block Diffusion Language Models (MBD-LMs) to enhance the decoding parallelism and throughput of diffusion-based text generation. This approach, which includes Multi-block Teacher Forcing post-training and an optimized Block Buffer inference engine, increased Tokens Per Second by up to 79.19% while also improving generation accuracy on models like LLaDA2-Mini.

Key insights

  • Explicitly aligning the model's training states with the heterogeneous, bounded noisy block patterns of practical multi-block inference is crucial to achieve parallel decoding without degrading accuracy.
  • A chain-uniform noise-scheduling strategy during training effectively creates the necessary diverse and separated slot-wise noise gaps required for robust MultiBD performance, unlike simpler monotonic schedules.
  • System-level engineering, particularly maintaining static input shapes via mechanisms like the Block Buffer, is vital for leveraging hardware optimizations (e.g., CUDA Graphs) and realizing wall-clock speedups from increased decoding parallelism.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.