Reproduce “Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhihao Guo, Zonghan Wu, Haizhou Du, Huan Huo, Yilei Shao, Athanasios V. Vasilakos, Qingsong Wen
Looped Transformers can sometimes propose a useful update whose full application overshoots the region of improving task utility. By analyzing utility along fractional recurrent steps, the study shows that local curvature predicts these finite-step failures and that adaptive step contraction can recover lost progress, including 94.2% of identified failures for Ouro-1.4B on GSM8K.
Key insights
- Negative effects from extra recurrence are not always directional failures: some updates initially improve reference utility but become harmful only because the model moves too far.
- Curvature is essential for forecasting full-step outcomes; the quadratic predictor achieves 97.19%–99.92% sign accuracy and reduces mean absolute error by roughly 5.9–29.8 times over first-order prediction.
- Intermediate states can outperform both the current and full-step states, although the benefit varies substantially by model, task, and recurrent transition.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.