Reproduce “Training Trajectories of Language Models Across Scales” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, Ves Stoyanov

Mengzhou Xia et al. systematically analyzed the training trajectories of large language models across scales, revealing that validation perplexity serves as a consistent predictor for model behaviors and in-context learning capabilities. They also demonstrated that tasks previously considered to show "emergent abilities" exhibit smooth, continuous performance growth when observed over intermediate checkpoints.

Key insights

  • Validation perplexity serves as a robust predictor for language model behaviors and downstream task performance, often correlating more strongly than model size or training computation.
  • The study challenges the notion of abrupt "emergent abilities," showing that performance improvements in complex downstream tasks appear as smooth, continuous growth when observed across intermediate training checkpoints.
  • Systematic generation of texts favored by smaller models through interpolated decoding reveals that while smaller models acquire linguistic competence, larger models further refine their understanding by assigning lower probabilities to semantically unlikely or factually inconsistent sequences.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.