Reproduce “Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison” on OpenResearch

About this paper

Caio Vicentino

A controlled comparison of autoregressive (AR) and masked diffusion language models (MDLM) on identical data and compute budgets reveals that MDLM offers significantly higher structural diversity in generated text, especially at narrative beginnings, at a comparable training throughput to AR models, which tend to overfit earlier.

Key insights

• Diffusion model training throughput is nearly identical to autoregressive models (MDLM at 95.5% of AR's speed), challenging the assumption that diffusion is substantially more expensive to train.
• AR models exhibit rapid convergence and early overfitting, reaching optimal performance around 14,000 steps, whereas MDLM shows slower, continuous improvement without overfitting even after 20,000 steps, suggesting different optimal training durations.
• A clear diversity-fluency trade-off exists: AR models produce highly fluent but structurally repetitive text due to prefix mode collapse, while MDLM generates significantly more diverse narratives, particularly in openings, sometimes at the cost of minor grammatical inconsistencies.

Summary by alphaXiv.