Reproduce “Almost Free State Prediction Separation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
John Langford, Nathan Godey, Giovanni Monea, Yoav Artzi, Harry Dong, Ying Fan, Gustavo de Rosa, Zheng Zhan
The paper presents a position-free prediction stream for autoregressive Transformers. A weight-shared stream starts from pause embeddings, queries the state stream’s keys and values, and writes no keys or values. On a 1B model, a phased version reached 2.8691 cross-entropy versus 2.8957 for control at 100B tokens, and improved by 0.012 nats at equal node-hours; evidence is limited to one scale and seed.
Key insights
- Separating persistent contextual representations from prediction-specific representations improves language modeling without adding model parameters, token positions, or KV-cache entries.
- Most of the loss improvement appears within roughly 15B tokens after activation, so late-stage adoption can capture much of the benefit at lower cost.
- The practical value depends strongly on implementation and accounting: phased variants retain gains under equal wall-clock compute, while improvements are smaller under strict equal-FLOP comparisons.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.