Reproduce “Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
James Town, Etienne Boursier, Ben Lewis, Matthias Englert, Ranko Lazic
The authors characterize the limiting training dynamics of a two-layer ReLU network on orthonormal regression data. With balanced, vanishingly small initialization, training proceeds through saddle-to-saddle jumps, each activating one neuron to fit still-unfitted examples on which it is active. The resulting interpolator has squared parameter norm of order √n under specified label conditions, but is not generally minimum-norm.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.