Reproduce “LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fengqi Zhu, Shaoxuan Xu, Jingyang Ou, Zebin You, Yipeng Xing, Huabin Liu, Xiaolu Zhang, Jun Zhou and 6 more
The authors develop scaling guidance for mixture-of-experts diffusion language models by recalibrating training choices rather than copying autoregressive recipes. Sweeps show that optimal training tokens grow faster than activated computation, and favor lower activation ratios at larger scales. Using this guidance, 30B-A3B LLaDA MoE v2, trained on 23.5T tokens, beats its predecessor on several benchmarks at lower training FLOPs, not inference cost.
Key insights
- Optimal batch size for MoE dLLMs grows more steeply (B* = 0.374 * C^0.3481) and learning rate decays faster (η* = 64.8 * C^-0.2447) with compute compared to AR models, benefiting from larger nominal batches due to masked denoising.
- Optimal compute allocation exhibits a data-side tilt (D* ∝ C^0.525, M* ∝ C^0.475), implying that marginal compute is best invested relatively more in additional training tokens for MoE dLLMs.
- Scaling MoE dLLMs favors lower activation ratios (sparser activation) at larger compute scales, benefits from moderate expert granularity (G=8-16), and consistently maintains an optimal shared-expert ratio of 33.3% across all compute scales.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.