Reproduce “Understanding the Role of Reasoning Data in Pretraining” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jianhao Huang, Allan Zhang, Ali Payani, Baharan Mirzasoleiman

The authors study how to divide a fixed training budget between atomic facts and multi-step reasoning traces. In controlled state-tracking experiments, longer chains generally favor more atomic supervision, while traces teach composition and atomic examples cover transitions omitted from traces. For Olmo-3 150M, an 80/20 FineMath/MathLake mixture achieved GSM8K solution-sequence cross-entropy 1.608, beating either pure source in this setup.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.