Reproduce “Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Louis Bethune, David Grangier, Dan Busbridge, Eleonora Gualdoni, Marco Cuturi, Pierre Ablin
This study establishes quantitative scaling laws that predict overfitting and catastrophic forgetting during Large Language Model finetuning. It demonstrates that injecting as little as 1% of pretraining data effectively mitigates forgetting without significantly impacting target task performance.
Key insights
- Injecting a remarkably small fraction of pretraining data, as little as 1%, is sufficient to substantially prevent catastrophic forgetting while maintaining or improving generalization on the target domain.
- A novel scaling law for forgetting quantifies the efficiency of pretraining data injection (via a `Bp` factor), showing its strong and domain-dependent effect on retaining general knowledge.
- The derived scaling laws exhibit strong extrapolation capabilities, allowing accurate prediction of performance for larger models and datasets based on results from smaller-scale experiments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.