Reproduce “Improving Pretraining Data Using Perplexity Correlations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tristan Thrush, Christopher Potts, Tatsunori Hashimoto

A method for efficiently selecting high-quality pretraining data for large language models is presented, which utilizes perplexity correlations from a diverse set of existing public LLMs to identify beneficial domains without requiring new model training. This approach outperforms other training-free baselines, performs comparably to robust hand-crafted filters, and shows increasing benefits for models scaled up to 1.4 billion parameters trained on 28.8 billion tokens.

Key insights

  • Publicly available LLMs can act as 'sensors' to assess the quality of pretraining data by measuring their perplexity on different web domains.
  • A principled statistical framework, using rank-based correlation, can estimate the relationship between an LLM's perplexity on specific data domains and its downstream benchmark performance.
  • This estimated perplexity-benchmark correlation can guide a training-free data selection process, identifying valuable data subsets without needing to retrain new language models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.