Reproduce “A Hitchhiker's Guide to Scaling Law Estimation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Leshem Choshen, Yang Zhang, Jacob Andreas
This paper offers a practical guide for estimating Large Language Model (LLM) scaling laws, based on a large-scale meta-analysis of 485 models across over 40 families. It demonstrates how to achieve reliable predictions (typical 4% Mean Absolute Relative Error) by efficiently leveraging intermediate training checkpoints and diversifying preliminary experiments, providing data-driven recommendations for cost-effective LLM development.
Key insights
- Intermediate training checkpoints are highly valuable for fitting scaling laws, improving prediction accuracy over using only final loss values.
- Scaling law parameters differ significantly across model families, yet in some cases, performance can be accurately estimated by borrowing parameters or training a target model for only a fraction of its full run.
- The 5 parameters of the Hoffmann et al. scaling law empirically exhibit fewer degrees of freedom than assumed, suggesting potential for simpler functional forms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.