Reproduce “Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen, Zhengzhong Liu, Eric Xing, Xuezhe Ma
The authors study looped language models that reuse Transformer blocks, encoding the prompt once, refining it through a middle block with prompt access at each iteration, then decoding. At 500B tokens, Huginn uses 8.0B resident parameters versus 32.0B for a matched-FLOP feedforward MoE and scores 59.65 versus 56.81 F1 on DROP; this comparison concerns resident weights and performance, not speed.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.