Reproduce “UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cheng Liang, Tengfei Zhang, Ya Zhang, Yanfeng Wang, Zhouhan Lin
UpLoop reallocates pretraining compute across stages: first training shallowly, then copying learned layers to grow depth, and finally repeating a shared middle block. It matches a dense baseline’s pretraining tokens and analytical FLOPs. At 1.3B, its equally weighted 0–100 score across 22 eligible tasks rises from 22.99 to 27.52; matching these budgets does not establish equal elapsed time or inference cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.