Reproduce “Functional Bilevel Optimization for Machine Learning” on OpenResearch

About this paper

Ieva Petrulionyte, Julien Mairal, Michael Arbel

FuncID defines bilevel learning over prediction functions rather than neural-network weights, using a unique function-level inner solution under strong convexity. It fits prediction and adjoint functions with parameterized models to estimate outer gradients. On dSprites, it outperforms DFIV with 5,000 training samples but is comparable at 10,000; these results concern tested settings, and its efficiency comparison is per-gradient, not end-to-end.

Key insights

• Optimizing in function space rather than parameter space allows for stronger theoretical guarantees, as strong convexity of the inner objective with respect to the prediction function is often satisfied, ensuring a unique inner solution.
• The functional adjoint sensitivity method enables the computation of the total gradient by minimizing a strongly convex quadratic objective in the Hilbert space, providing a stable and well-conditioned problem.
• FuncID's approach to computing Hessian-vector products with respect to the output dimension of the prediction function (typically small) offers significant memory and time efficiency compared to parametric methods that compute with respect to large model parameter dimensions.

Summary by alphaXiv.