Reproduce “Data Schedules and Learning Rate Schedules Should Be Designed Together” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haodong Wen, Kairong Luo, Tingkai Yan, Wenguang Chen, Kaifeng Lyu
The authors study how mid-training (MT) data allocation interacts with learning-rate (LR) schedules under fixed compute and MT-token budgets. They compare curated mathematics with pretraining data. The preferred allocation reverses: mixing PT and MT late performs best under full-range decay, while an all-MT ending performs best under constant LR. Delayed, short decay gives the lowest tested validation loss, not a proven global optimum or downstream gain.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.