Reproduce “CausalMix: Data Mixture as Causal Inference for Language Model Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zinan Tang, Yukun Zhang, Shaomian Zheng, Zhuoshi Pan, Qizhi Pei, Dingnan Jin, Qing Cui, Jun Zhou and 2 more
CausalMix selects domain mixtures for supervised fine-tuning by estimating how mixture changes relate to downstream performance at measured data-pool states. It uses historical proxy runs, pool characteristics, and an orthogonalized model of mixture and outcome. On Qwen2.5-7B trained on 800K Tulu 3 examples, CausalMix-S reports the highest development-set Avg, 62.28, but its unseen-set Avg Uns, 47.98, is not highest.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.