Reproduce “Towards Looped Models Done Right, Part II: Rethinking at Fixed Points” on OpenResearch

About this paper

Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen, Zhengzhong Liu, Eric Xing, Xuezhe Ma

The authors train looped language models for terminal-state prediction and KV-cache sharing. A learned depth prior allocates supervision across recurrence counts, while orthogonal input injection preserves the injected signal. At 1.6B parameters and five recurrences, the model uses four terminal KV banks and matches a full-cache fixed-depth model’s seven-task average, 60.00% versus 59.90%. The study uses one seed per configuration.

Key insights

• Fixed-point behavior can connect efficiency techniques across the model lifecycle, but these savings depend on training recurrent states to become sufficiently settled.
• Depth supervision affects whether terminal KV sharing remains accurate: an adaptive, entropy-regularized depth prior performs better with sharing than training only at one fixed depth.
• OrthoInj controls the strength of repeated input injection by construction; the tested baseline instead amplified the aligned input component at the fixed point.

Summary by alphaXiv.