Reproduce “Looping Deeper Without Growing the KV Cache: Recurrent Depth for Causal Encoder-Decoders” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiaqi Leng, Minshen Zhang, Zhaofeng Sun, Yu-Yang Qian, Peng Zhao, Hao Zhang

The authors introduce a recurrent encoder loop for causal encoder–decoder models that reuses a layer block while sharing final-round global attention key/value states across rounds. Stochastic-depth training supports this cache sharing in tested models. At 150M parameters, encoder looping achieves 0.8468 validation BPB versus 0.8648 for decoder looping; at 552B, serving fits 38 128K requests, but model quality was not tested.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.