Reproduce “Trained Transformers Learn Linear Models In-Context” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ruiqi Zhang, Spencer Frei, Peter L. Bartlett

Research from UC Berkeley rigorously demonstrates how gradient-based optimization enables transformers to learn in-context, showing they effectively mimic optimal linear predictors like OLS. The study quantifies prediction error scaling with context length and clarifies the models' robustness to task and query shifts versus brittleness to covariate shifts.

Key insights

  • Gradient flow can globally converge to an optimal solution for training transformers on in-context learning tasks, despite the non-convex objective.
  • Trained transformers implicitly learn an algorithm that closely approximates the best linear predictor (Ordinary Least Squares) from in-context examples.
  • The prediction error for in-context learning scales as O(1/M + 1/N^2), where M is the number of test examples and N is the training context length, indicating an inherent limitation for finite N.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.