Reproduce “Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shane Bergsma, Nolan Dey, Joel Hestness

The authors introduce a training re-evaluation curve (TREC) that measures final-model loss on batches by training position, then predict its shape from AdamW dynamics to guide data placement. In tested settings, placing high-quality data at the TREC minimum improved target validation loss over end placement in a 3.9B continual-pretraining test; it did not reliably select optimizer configurations or ensure gains on other tasks.

Key insights

  • Training Re-evaluation Curves (TRECs) accurately predict the optimal placement of high-quality data within a training curriculum for maximizing model performance.
  • The AdamW optimizer's implicit Exponentially Weighted Moving Average (EMA) timescale is the primary factor determining the shape of TRECs, overshadowing the influence of momentum (β1) and velocity (β2) parameters.
  • A training-fraction-dependent exponent is essential for accurately modeling TREC shapes, reflecting how minimizer drift influences the retention of early training data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.