Reproduce “Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sean McLeish, Ang Li, John Kirchenbauer, Dayal Singh Kalra, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Jonas Geiping and 2 more

Research from the University of Maryland and collaborators introduces a method to convert existing pretrained language models into depth-recurrent architectures, enabling them to "think deeper" at test time. This approach efficiently enhances reasoning capabilities on tasks like mathematics while preserving general language modeling performance, outperforming traditional post-training for the same computational budget.

Key insights

  • Initializing recurrent models with pretrained fixed-depth weights drastically improves training efficiency and final performance over random initialization.
  • A progressive recurrence scheduling curriculum during training is critical for managing computational costs and enhancing model stability.
  • Careful layer selection for the recurrent block combined with a data 'healing' phase successfully maintains broad language modeling capabilities post-architectural modification.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.