Reproduce “First-Order Preconditioning via Hypergradient Descent” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ted Moskovitz, Rui Wang, Janice Lan, Sanyam Kapoor, Thomas Miconi, Jason Yosinski, Aditya Rawal

The FOP method learns an online, layer-specific transformation of gradients rather than explicitly computing an inverse-Hessian or inverse-Fisher preconditioner. It can rescale and mix directions using first-order information. On a 9-layer CIFAR-10 CNN, FOP reached 91.5 ± 0.2 test accuracy, versus 90.9 ± 0.1 for momentum, adding 65 parameters and 0.8% reported overhead, on tested settings only.

Key insights

  • Second-order-like benefits, such as gradient rotation and scaling, can be achieved by learning a flexible preconditioning matrix using only first-order information via hypergradient descent, bypassing explicit Hessian or Fisher matrix computations.
  • The learned preconditioning matrices develop a 'whitening' or decorrelating structure in the learning signal, characterized by strong positive diagonal elements and negative weights to immediate neighbors.
  • FOP enhances the robustness of standard optimizers to hyperparameter choices, significantly improving performance across a wider array of learning rates and momentum coefficients.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.