Reproduce “ARLoop: Fully-Open Pre-trained Looped Transformers with Attention Residuals” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuekun Yao, Yupei Du, Dawei Zhu, Huan Sun
ARLoop lets each sublayer attend over summaries of earlier depth, selecting among them rather than relying on one accumulated residual stream. In controlled experiments, sliding-window attention at 370M and three recurrences achieved 12.17 validation perplexity and 50.67% downstream accuracy, versus 12.27 and 50.23% for full-history attention. This pattern does not hold in every setting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.