Reproduce “Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Binghua Li, Mengzhe Li, Denny Wu, Tianhao Wang
The authors study how a two-neuron ReLU student learns two ReLU teacher units under population gradient flow with isotropic Gaussian inputs. They classify stationary points and trace the dynamics: both student units first approach the teachers’ bisector, then specialize. For acute teacher angles, nonzero-weight stationary points are global minima or saddles; logarithmic escape-time results require structured initialization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.