Reproduce “LLaDA2.0: Scaling Up Diffusion Language Models to 100B” on OpenResearch

About this paper

Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu and 23 more

The LLaDA2.0 system converts pretrained autoregressive models into block diffusion models through warmup, full-sequence training, and block-size decay. This staged schedule adapts next-token predictors to denoise masked text, then supports blockwise inference. Its 100B-total-parameter flash model reports an aggregate score of 73.18 versus 73.60 for Qwen3-30B-A3B-Instruct-2507, while leading on some coding benchmarks but trailing on others.

Key insights

• Discrete diffusion language models can effectively scale to 100 billion parameters by leveraging knowledge transfer from pre-trained autoregressive models, challenging the AR paradigm's dominance.
• The Warmup-Stable-Decay strategy successfully adapts AR models to the diffusion framework, bridging architectural and inductive bias differences across different generation paradigms.
• Diffusion models exhibit inherent strengths in parallel generation, structured tasks like coding and agent capabilities, and can achieve significant inference speedups over autoregressive counterparts.

Summary by alphaXiv.