Reproduce “Conversion and Scaling Laws for Autoregressive and Masked Diffusion Language Models on Shared Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Oliver Wiggins-Hay
The authors compare compute allocation across AR model sizes and data amounts, and measure conversion to masked diffusion language models (MDLMs). A coupled loss model captures larger AR models’ greater gains from added data and forecasts two 2.9B-model losses within 1 millinat. Conversion saves diffusion tokens at matched MDLM loss, but total savings require short AR training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.