Reproduce “In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haotian Gu, Yizhou Xu, Lenka Zdeborová

The paper compares two linear-attention learners for predicting nonlinear tasks: one uses fixed nonlinear features before attention, while the other attends to raw inputs and learns a nonlinear readout afterward. In the population limit, the fixed-feature learner needs context of order d^k for degree k, while the feature learner transitions at order d. Finite-data errors differ for seen versus fresh tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.