Reproduce “Test-time regression: a unifying framework for designing sequence models with associative memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ke Alexander Wang, Jiaxin Shi, Emily B. Fox
The authors present a framework that treats sequence layers as fitting regression functions to stored associations, then querying them with inputs. It connects choices about memory weighting, mapping class, and optimization to architectures including linear and softmax attention. In synthetic recall tests, one regression layer solves the task when memory capacity suffices; this finding is limited to the tested stationary setup.
Key insights
- Linear attention's limitations stem from ignoring key covariance in its implicit linear least squares solution.
- Query-key normalization (QKNorm) in softmax attention is crucial for it to function as a locally constant nonparametric regressor.
- Fast-weight networks and online learners implicitly perform first-order optimization (e.g., SGD) on a least-squares objective during inference.
- The framework allows for principled derivation of novel architectures, like higher-order generalizations of softmax attention.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.