Reproduce “Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Grigory Sapunov
Learned memory tokens are empirically necessary for Universal Transformers to perform complex recursive reasoning tasks, with a model achieving 57.4% exact-match on Sudoku-Extreme. This work also presents a 'deep-start' initialization for Adaptive Computation Time, which resolves common training failures and enhances model reliability and efficiency.
Key insights
- Memory tokens are empirically essential for a minimal single-block Universal Transformer to achieve non-trivial performance on complex reasoning tasks, indicating a fundamental bandwidth limitation of the residual stream alone.
- The 'shallow-halt trap' in ACT-enabled models is an optimization pathology that can be reliably resolved by a novel 'deep-start' initialization, significantly improving training stability and success rates.
- Memory tokens and ponder depth function as substitutable computational resources, allowing models to achieve comparable accuracy with fewer steps by leveraging more memory capacity when under computational pressure.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.