Reproduce “It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs” on OpenResearch

About this paper

Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov, Carlos Rosas-Hinostroza, Neil Si Smail, Benjamin Burtin, Hanna Shcharbakova and 2 more

SYNTH trains small language models in one stage on synthetic questions, grounded answers, and reasoning traces generated from controlled factual seed passages. It varies tasks and uses retrieved contexts instead of relying first on a web crawl. In a 600M comparison, it scores 42.2% versus 25.1% on 21 multiple-choice tasks excluding MMLU, but factual recall remains bounded by seed coverage.

Key insights

• Reasoning traces have limited impact on multiple-choice accuracy but improve open-ended performance, especially truthfulness and domain reasoning; removing them lowers open-ended accuracy from 24.3% to 22.0%.
• The models learn high precision on knowledge represented in the seed corpus, but coverage remains narrow: the 600M MoE abstains on 66.8% of held-out entities and achieves 61.7% precision on attempted held-out answers.
• Synthetic data is effective for targeted adaptation: the same approach can teach a 600M model telecommunications knowledge and tool calling from relatively small domain-specific seed collections.

Summary by alphaXiv.