Reproduce “Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenyu Du, Tongxu Luo, Zihan Qiu, Zeyu Huang, Yikang Shen, Reynold Cheng, Yike Guo, Jie Fu

The paper evaluates model growth for LLM pre-training by training a smaller model, copying its layer sequence into a deeper target, and continuing training. Its Gstack operator is compared with other initialization methods in depth- and width-growth tests. For 7B models, Gstack reaches the scratch model’s loss at 194B rather than 300B tokens, a 54.6% FLOP-based speedup, without uniformly better quality.

Key insights

  • Depth-wise direct stacking (G_stack) consistently outperforms other model growth operators for LLM pre-training, offering substantial computational speedups.
  • The acceleration benefits of G_stack are robust and sustained across increasing LLM sizes (up to 7B parameters) and extremely large training data volumes.
  • Strict function preservation is not essential for effective model growth; surprisingly, some controlled perturbation can improve performance, and a 'one-hop' stacking approach is superior to gradual methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.