Reproduce “You Don't Need Feature Learning to Do In-Context Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Joseph Turnbull, Jamie Simon
The authors study how fixed-feature learners generalize on an in-context linear-regression task despite training on a finite task pool. They argue kernel spectral bias learns simpler predictor components before higher-degree terms that favor memorization. In lazy random-features experiments, novel-task error falls below the memorizing oracle around 10^5–10^6 samples, then rises toward it; this result does not establish the pattern in language models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.