Reproduce “Scaling Laws for Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Danny Hernandez, Jared Kaplan, Tom Henighan, Sam McCandlish
OpenAI researchers establish a quantitative framework for transfer learning, introducing "Effective Data Transferred" (DT) to measure pre-training benefits and showing that DT follows predictable power-law relationships with fine-tuning data size and model parameters. This work quantifies transfer efficiency in a natural language to Python code setting, revealing distinct data regimes and the phenomenon of "ossification."
Key insights
- Effective data transfer follows predictable power-law relationships, offering a quantitative framework for understanding transfer learning scaling.
- The exponent "β" (approximately 0.38) in the power law for model parameters appears universal across different pre-training distributions, suggesting fundamental scaling properties.
- Transfer learning exhibits distinct "low-data" and "high-data" regimes; benefits diminish and "ossification" (performance degradation) can occur in the high-data regime, especially for smaller models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.