Reproduce “Multi-ResNets for Subspace Preconditioning in Constrained Optimization” on OpenResearch

About this paper

Merve Karakas, Christopher J. Williams, Emmanuel O. Balogun, Sadegh Sadeghi Tabas, Christian Brown, Nikhil Rao

Researchers from UCLA, University of Oxford, and Tapestry (Google) introduced MResOpt, a staged residual neural network architecture for constrained optimization that leverages lexicographic constraint ordering. This framework prioritizes constraint satisfaction, achieving 2-7x lower high-priority constraint violations and significantly reducing equality manifold drift compared to prior iterative methods on complex non-convex problems like AC Optimal Power Flow.

Key insights

• Lexicographic ordering of constraints induces a useful filtration over feasible sets, enabling domain-informed prioritized satisfaction and providing a 'safe fallback' to physically valid states.
• Re-completion (re-projection onto the equality manifold) after each correction step is critical for preserving equality manifold adherence in nonlinear settings, addressing the off-manifold drift issue.
• The staged residual architecture acts as a sequential preconditioning mechanism, which, in the infinite-width limit, can be theoretically shown to ensure sequential convergence and preserve inductive biases.

Summary by alphaXiv.