Reproduce “BiMix: A Bivariate Data Mixing Law for Language Model Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, Bolin Ding
BIMIX, developed by researchers at Alibaba Group, introduces a bivariate data mixing law that systematically models how both data domain proportions and total data volume jointly influence Large Language Model performance. This framework accurately predicts model validation loss across unseen data mixtures and training steps with less than 0.2% mean relative error, enabling more efficient optimization of pretraining data composition than prior methods.
Key insights
- LLM validation loss scales with domain proportion and training steps in a separable, multiplicative power-law relationship, allowing for a bivariate model.
- BIMIX can accurately extrapolate validation losses to unseen, scaled-up training steps and generalize to unseen data mixtures with high fidelity (mean R 2 > 0.97).
- Entropy-based measures, particularly Conditional Entropy, serve as effective, training-free proxies for identifying important data domains and guiding efficient data mixture construction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.