Reproduce “Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaju Wu, Yi Hu, Muhan Zhang
The paper tests whether changing the state entering a loop steers a frozen Transformer block toward different graph-walk transitions. In held-out graph tests, a token-wise map raises one-hop accuracy from 48.2% to 99.6%, while a two-hop map reaches 98.2%. Swapping attention patterns changes answers, supporting controllability but not showing that attention is the only cause or improving inference efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.