Reproduce “Scalable Nested Optimization for Deep Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jonathan Lorraine

Jonathan Peter Lorraine's Ph.D. thesis develops scalable gradient-based optimization algorithms for nested problems in deep learning, enabling the tuning of millions of hyperparameters and stabilizing differentiable game dynamics. The work introduces methods like implicit differentiation and complex momentum, achieving improved performance in tasks such as dataset distillation, learned data augmentation, and Generative Adversarial Network training.

Key insights

  • Gradient-based hyperparameter optimization for millions of parameters is feasible with constant memory cost by leveraging implicit differentiation and approximating inverse Hessian-vector products via Neumann series.
  • Generalizing momentum to complex values provides a robust first-order optimization method capable of stabilizing diverse game dynamics, from purely cooperative to purely adversarial, by addressing rotational updates.
  • Lyapunov exponents can be used to identify bifurcation points in differentiable games, offering a principled method to discover a wider range of diverse solutions and equilibria in multi-agent systems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.