Reproduce “Large Language Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin and 2 more
LLaDA tests whether language models need left-to-right generation by training an 8B masked diffusion model on 2.3 trillion tokens and 4.5 million instruction pairs. It fills masked positions iteratively, sometimes remasking predictions. On 496 Chinese-poem pairs, it scores 45.6 versus GPT-4o’s 34.3 for reversal, but 51.8 versus 82.7 forward, limiting the result to this task.
Key insights
- Generative modeling principles, rather than the autoregressive formulation, are the core drivers of large language model capabilities and scalability.
- Diffusion models can natively support intrinsic bidirectional reasoning, offering a principled solution to the 'reversal curse' observed in autoregressive models.
- A principled probabilistic formulation and standard LLM training pipeline enable masked diffusion models to achieve comparable performance to state-of-the-art autoregressive models at an unprecedented scale.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.