Reproduce “Predictive Data Selection: The Data That Predicts Is the Data That Teaches” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kashun Shum, Yuzhen Huang, Hongjian Zou, Qi Ding, Yixuan Liao, Xiaoxin Chen, Qian Liu, Junxian He
PRESELECT, a two-stage data selection framework from HKUST and Vivo AI Lab, efficiently curates pre-training data by identifying documents whose compression efficiency predicts model capabilities. This approach enables models to achieve superior performance with a 10x reduction in compute requirements compared to baselines.
Key insights
- Data on which a model's compression efficiency (normalized loss) *predicts* its downstream abilities also effectively *contributes* to learning those abilities.
- Using a limited number of models from the *same family* for predictive strength calculation mitigates evaluation noise that can arise from architectural differences or prompt sensitivities across diverse model families.
- A ranking-based correlation metric is more robust for quantifying document-level predictive strength than Pearson correlation, which can be sensitive to noise.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.