Reproduce “Loopback Transformer: Towards a Unified View of Residual Looped Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xuan Ouyang, Zefan Cai, Chenlong Zhang, Cheng Luo, Ke Wan, Junjie Hu

Loopback reduces Transformer parameters by reusing a shared stack and storing each block’s update separately from its input. It decays older updates and uses attention to allocate weight among them, while keeping the input on a separate path. At matched depth and token budgets, its 374M model uses 56.3% as many Transformer parameters as Standard and achieves lower validation loss, but does not show faster execution.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.