Reproduce “Exact-Solution Volume and Length Generalization in Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yijia Jessica Zhu, David Chiang
The authors measure exact-solution volume (NESV) as input length grows in a fixed-width, single-layer, single-head transformer. For binary INDEX, the positional model has Θ(n⁻³) volume, while the selectively normalized model has Θ(n⁻¹). At length 511, after training through 51, accuracy is 59.6% versus 85.1%; this does not show that volume predicts trained accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.