Reproduce “Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiyuan Li, Tianhao Wang, JasonD. Lee, Sanjeev Arora
The authors characterize when reparameterized training produces mirror flow in the original parameters under continuous-time gradient flow. A commuting, regular parametrization satisfying a flow-domain condition induces geometry determined by the parametrization and initialization, not the loss. For convergent interpolating regression trajectories, this geometry selects a minimum-regularizer solution among interpolants; Nash’s embedding theorem gives a reverse construction, but it is implicit.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.