Reproduce “Architectures and KV Replay for Depth-Recurrent Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lucia Licakova, Niklas Schmidinger, Lukas Hauzenberger, Sebastian Böck, Sepp Hochreiter
The authors introduce recurrent KV replay training for looped Transformers, where selected iterations write keys and values and later iterations reuse them while computing fresh queries. In a six-iteration model, stride-3 replay retains two KV groups, using 192 KiB per token analytically versus 576 KiB with six groups. CORE and WikiText results are close to full-cache results under this matched setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.