Reproduce “Loop Before You Leap: What Recurrent Depth Can and Cannot Buy in Data-Limited Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daichi Hotta, Yohei Kobashi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
The authors test whether reusing Transformer layers improves grammatical benchmark performance under a 100-million-word pretraining budget. A 12-layer stack applied four times scores 5.36 percentage points above an independent 48-layer model at matched inference compute in text-only training. Recurrence does not clearly reduce the multimodal penalty, and the estimate is too uncertain to establish no effect.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.