Reproduce “Recurrent Looped Transformer” on OpenResearch

About this paper

Yifan Zhang

RLT carries decoder’s evolving state through every prompt and response token, retaining the final output and layerwise SWA key–value caches instead of resetting at serving. Under matching parameters, causal computation, cache rules, and deterministic operations, prompt prefill and incremental processing produce the same fixed-history distribution. A parity evaluation reached 100% on 256-bit tasks, but recurrence was not isolated and efficiency was unmeasured.

Key insights

• The recurrent path has structural depth proportional to sequence length: processing t tokens with an L-layer decoder yields tL decoder-block evaluations, although each individual token still runs a fixed number of layers.
• Prompt prefill remains partly sequential because the encoder is parallelizable but the decoder must perform ordered recurrent transitions; therefore the design does not by itself establish lower latency or lower total computation than a conventional Transformer.
• Correct replay and full BPTT require accounting for every parameter-dependent state, including recurrent outputs, encoder memory, and decoder KV caches; detaching only one of these components does not remove all gradient paths.

Summary by alphaXiv.