Reproduce “Understanding Dynamic Compute Allocation in Recurrent Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ibraheem Muhammad Moosa, Suhas Lohit, Ye Wang, Moitreya Chatterjee, Wenpeng Yin

This research from Pennsylvania State University and Mitsubishi Electric Research Lab introduces a controlled evaluation framework to investigate if dynamic compute allocation in recurrent Transformers truly aligns with token-level computational complexity. It found that while models can learn to adjust computation based on difficulty without explicit supervision, this ability does not directly translate to improved algorithmic generalization or extrapolation.

Key insights

  • Complexity-aligned compute allocation can emerge implicitly in recurrent Transformers, without explicit supervision for token-level difficulty, driven by cross-entropy loss and generic compute regularization.
  • While models can align compute with task complexity, this alignment does not automatically lead to improved algorithmic generalization or extrapolation capabilities to unseen input sizes, indicating models often learn size-specific correlations rather than robust algorithms.
  • The timing of compute allocation decisions (early vs. online) fundamentally shapes the learned policy, with early decisions relying on static input cues and online decisions tracking dynamic algorithmic execution states.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.