Reproduce “Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Stanislas Ducotterd, Alexis Goujon, Pakshal Bohra, Dimitris Perdios, Sebastian Neumayer, Michael Unser
The authors develop learned linear spline (LLS) activations for 1-Lipschitz neural networks and test them in plug-and-play MRI reconstruction. Each neuron learns a piecewise-linear response while slope constraints preserve the Lipschitz bound. On single-coil fastMRI at setting (4, 0.08), LLS scores 31.54 PSNR and 0.7924 SSIM, above absolute value’s 30.61 and 0.7721; this compares trained systems under the paper’s protocol.
Key insights
- 1-Lipschitz networks employing learnable linear splines represent global optima of a constrained functional optimization problem incorporating second-order total-variation regularization, providing a strong theoretical foundation.
- The "SplineProj" method provides a practical and differentiable approach to strictly maintain 1-Lipschitz constraints on complex learnable activation functions, addressing a key implementation challenge.
- Even under strict 1-Lipschitz constraints, LLS activation functions achieve significantly greater expressivity than prior methods, enabling superior performance in demanding applications like image reconstruction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.