Reproduce “Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peilong Zhou, Hongtao Zhang, Yulong Huang, Cangyuan Li, Mingyu Chen, Ying Wang
Authors test whether looped transformers compute, retain, and safely reuse intermediate states in in-context Mealy-machine executions, using simulator-provided targets at every position and for counterfactual inputs. A hidden-state swap redirects 237 eligible continuations exactly in ten models, but freezing a state when its answer first becomes readable damages later accuracy; delaying freezing restores accuracy in tested models, not a universal schedule.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.