Reproduce “How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan, Shruti Bhosale, Mike Lewis, Maryam Fazel-Zarandi, Shashank Rajput, Sujan Kumar Gonugondla
The authors study looping, which reuses transformer weights for repeated computation, in dense decoder-only models. They compare loop designs and shifted parallel looping. In tested over-trained conditions, two-loop models at 80 tokens per parameter reach a given DCLM validation bits-per-byte with fewer training FLOPs than vanilla models at 160 tokens per parameter, without establishing general efficiency or latency gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.