Reproduce “YuLan-Mini: An Open Data-efficient Language Model” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yiwen Hu, Huatong Song, Jia Deng, Jiapeng Wang, Jie Chen, Kun Zhou, Yutao Zhu, Jinhao Jiang and 3 more

YuLan-Mini, developed by the Gaoling School of Artificial Intelligence at Renmin University of China, introduces a 2.42 billion parameter open language model that achieves competitive performance using only 1.08 trillion training tokens. The model demonstrates strong capabilities in mathematical reasoning and code generation, attributed to an elaborate data pipeline and robust optimization strategies.

Key insights

  • Competitive LLMs can be developed with significantly less data (1.08T tokens) through meticulous data engineering and robust optimization, challenging the "bigger is better" paradigm.
  • Systematically addressing training instability via methods like μP-like initialization and WeSaR re-parametrization is crucial for efficient and reliable large-scale model training.
  • Strategic generation and integration of high-quality synthetic reasoning data can endow smaller models with advanced capabilities in specialized domains like mathematics and coding.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.