Reproduce “Solvable dynamics model of in-context learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruoran Bai, Haiping Huang
The authors solve training dynamics for in-context linear regression in a finite-data, single-layer, single-head linear-attention model. They derive exact weight trajectories for the quadratic empirical objective. With D=10, N=100, and weight decay 0.01, test loss rises for P=10 but falls for P=100, while training loss decreases in both cases. This contrast is limited to the specified model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.