Reproduce “LLaDA-MoE: A Sparse MoE Diffusion Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fengqi Zhu, Zebin You, Yipeng Xing, Zenan Huang, Lin Liu, Yihong Zhuang, Guoshan Lu, Kangyu Wang and 18 more
A research team from Ant Group and collaborating universities developed LLaDA-MoE, a Masked Diffusion Model incorporating a sparse Mixture-of-Experts architecture, marking the first successful pretraining of such a model from scratch. LLaDA-MoE-7B-A1B-Instruct, activating only 1.4 billion parameters, achieves performance comparable to a dense 3-billion-parameter autoregressive model, outperforming existing diffusion models.
Key insights
- A sparse Mixture-of-Experts architecture can be successfully integrated into and pretrained within a Masked Diffusion Model framework, validating a new direction for efficient language model design.
- The use of auxiliary losses for load balancing and router regularization is critical for stabilizing expert routing during the extensive pretraining of MoE-based diffusion models.
- Diffusion models, when enhanced with MoE, can achieve computational efficiency benefits similar to those observed in autoregressive MoE models, offering a path to high-performance models with reduced inference costs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.