Reproduce “Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhiwei Xu, Shihao Wu, Hanseul Cho, Wei Hu, Yixin Wang

MIR adds an auxiliary masked-input next-token loss to autoregressive training, preserving the architecture and inference interface. On DCLM with 100M unique tokens, it lowered validation loss across five model sizes from 72M to 1.4B, reaching 3.317 versus 3.347 at 1.4B. SoftQ improved held-out scaling-loss prediction, but evidence is limited to 1.4B parameters and 400M unique tokens.

Key insights

  • Strong weight decay is crucial for effective regularization in data-constrained settings for both autoregressive and masked diffusion language models, suggesting that prior dLLM advantages were partly due to under-regularized AR baselines.
  • The marginal value of additional unique data increases with model size in data-constrained regimes, highlighting a misspecification in classical additive scaling laws that predict a constant loss gap.
  • Random input masking provides additional regularization benefits to autoregressive models, even when strong weight decay is already applied, improving robustness to noisy or unusual local prefix contexts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.