Reproduce “PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Oskar van der Wal, Pietro Lesci, Max Muller-Eberstein, Naomi Saphra, Hailey Schoelkopf, Willem Zuidema, Stella Biderman

Researchers introduced PolyPythias, an expanded dataset of 50 multi-seed, multi-size language model pre-training runs with 7,000 checkpoints, extending the EleutherAI Pythia suite. The work systematically investigates the stability of decoder-only LMs across different initial conditions, revealing largely stable pre-training dynamics, consistent learning phases, and methods for identifying outlier runs through training maps which can predict final model performance with high accuracy (R^2 > 0.9) even across model sizes.

Key insights

  • LM pre-training dynamics are largely stable across different random seeds and model sizes, following consistent scaling laws for both downstream performance and internal representational properties.
  • A critical learning landmark occurs around 10^3 steps (2.5B tokens) where models begin generating non-random answers, acquire core semantic functions, and show initial improvements in representational efficiency.
  • "Training maps," derived from HMMs on model parameter statistics, effectively diagnose training stability, identify outlier runs characterized by "forks" or regressions to earlier states, and provide insights into drivers of state transitions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.