Reproduce “Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
A. Bochkov
The authors study whether a model can grow while keeping a bounded set of parameters trainable. They train new Transformer blocks above a fixed token interface, freeze earlier dense blocks, and update only the newest blocks and output head. In a 9-layer comparison, growth uses 105.0M trainable parameters versus 180.5M, with higher final training perplexity: 4.12 versus 3.71.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.