Reproduce “Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tyler A. Chang, Zhuowen Tu, Benjamin K. Bergen

A study from UC San Diego characterized the convergence patterns of individual tokens in context during language model pre-training, introducing metrics for learning speed, 'forgetting' (increases in surprisal), and prediction stability. The research found that these fine-grained learning dynamics are systematically influenced by distributional statistics such as token frequency and n-gram probability.

Key insights

  • Sudden learning and 'forgetting' events for individual examples are surprisingly consistent across different pre-training runs, suggesting systematic rather than purely stochastic changes in model processing.
  • Language model learning proceeds in a sequential manner, beginning with the robust acquisition of high-probability n-grams, followed by a refinement phase for low-probability n-grams and the utilization of longer contexts.
  • Aggregate loss curves can obscure significant fluctuations and instability for individual, 'tail' examples, meaning a model appearing converged at an aggregate level may still be highly variable for specific, low-probability n-grams.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.