Reproduce “Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lior Belenki, Alekh Agarwal, Tianze Shi, Kristina Toutanova
Google DeepMind and Google Research developed a method to optimize large language model pre-training data mixtures, introducing the Mixture of Data Experts (MDE) approximation. This approach enhances the accuracy and sample efficiency of regression-based mixture optimization, yielding models with superior generalization on validation sets and improved performance across downstream tasks.
Key insights
- The Mixture of Data Experts (MDE) significantly improves the accuracy and sample efficiency of regression-based data mixture optimization, particularly in low-data regimes.
- Optimizing for validation losses on end-task-relevant domains strongly correlates with improved downstream task performance, unlike optimizing solely for pre-training domain perplexity.
- The method provides a theoretical justification for aggregating data expert predictions as an efficient approximation for models trained on data mixtures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.