Reproduce “Extrapolating from Regularised Solutions for Solving Ill-Conditioned Linear Systems in Machine Learning” on OpenResearch

About this paper

Disha Hegde, Jon Cockayne, Chris. J. Oates

Autonugget stabilizes ill-conditioned linear solves while preserving differentiability. It solves at several nugget values, then uses polynomial Richardson extrapolation to estimate the zero-nugget solution, with adaptive nugget selection and a custom JAX differentiation rule. Across 100 ill-conditioned matrices, its mean absolute error was 1.80 × 10⁻⁴, below LU-adapt’s 2.31 × 10⁻⁴ but above TSVD’s 7.12 × 10⁻⁶.

Key insights

• Richardson extrapolation can significantly accelerate the convergence of Tikhonov-regularized solutions to the unregularized solution, theoretically improving the error rate from O(h) to O(h^n) in exact arithmetic.
• An adaptive selection mechanism for nugget scaling (h) is crucial in finite precision arithmetic, effectively balancing numerical stability from larger nuggets with reduced extrapolation bias from smaller ones.
• Integrating a custom automatic differentiation rule ensures that adaptive regularization schemes remain compatible with modern ML training workflows, overcoming a critical barrier for practical application.

Summary by alphaXiv.