Reproduce “Always Learning, Always Mixing: Efficient and Simple Data Mixing All the Time” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho, Tal Linzen, Pratyusha Sharma

OP-Mix selects training-data mixtures across pretraining, continual midtraining, and instruction tuning using low-rank adapters trained from the current model to estimate candidate mixtures, then trains on the selected mixture. On Qwen2.5-7B-Instruct, SFT with OP-Mix reached 60.0% mean accuracy versus SDFT’s 60.2%, with 95% less compute; evidence is limited to tested scales and domains.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.