Reproduce “A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Behrad Moniri, Donghwan Lee, Hamed Hassani, Edgar Dobriban
The authors analyze one-step feature learning for a Gaussian single-index teacher. A growing gradient step updates the first layer, and ridge regression fits the second layer using separate data. Step size determines whether the representation gains a linear direction or also a quadratic one. In this asymptotic regime, the quadratic direction improves training and test errors when its coefficient is nonzero.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.