Reproduce “Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiawen Tao, Miao Peng, Yaoming Li, Xiaokun Yuan, Mengzhou Wu, Wenhan Yu, Guoan Wang, Nuo Chen and 2 more
The authors turn retrieved material into books by clustering passages, planning a contents hierarchy, and generating source-grounded sections in order. They compare intact books with the same sections trained separately in matched mid-training. On an in-house MoE model, intact books improve the unweighted mean across 28 benchmarks by 1.02 points; this supports preserving structure in this setup, but does not identify its cause.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.