Reproduce “From Growing to Looping: A Unified View of Iterative Computation in LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ferdinand Kapl, Emmanouil Angelis, Kaitlin Maile, Johannes von Oswald, Stefan Bauer

This research unifies depth-grown and looped Large Language Models by demonstrating they share an underlying mechanism of iterative computation, which enhances reasoning capabilities. The study proposes a "grow first, loop later" strategy, where depth-grown models show up to a 2x improvement in reasoning primitive accuracy when subjected to inference-time looping without explicit prior training for recurrence.

Key insights

  • Both looped and depth-grown models exhibit convergent internal computational signatures, such as increased reliance on later layers and periodic (sub)layer usage, suggesting a shared inductive bias towards iterative refinement.
  • Depth-grown models, despite not being explicitly trained for recurrence, demonstrate a robust ability to benefit from inference-time looping, significantly improving reasoning.
  • A hybrid design principle, combining unique outer layers with a recurrent middle block, provides robust recurrence while balancing stability and iterative benefits.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.