Reproduce “Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shaik Basheeruddin Shah, Pradyumna Pradhan, Wei Pu, Ramunaidu Randhi, Miguel R. D. Rodrigues, Yonina C. Eldar
This research provides optimization guarantees for finite-layer unfolded networks like LISTA and ADMM-CSNet in an over-parameterized regime. By extending the Polyak-Łojasiewicz (PL*) condition and using smooth soft-thresholding, the work shows these structured networks achieve near-zero training loss and can memorize more training samples compared to standard feed-forward networks, while maintaining superior parameter efficiency.
Key insights
- The Hessian spectral norm for unfolded networks scales with the square root of the network's width (`m`), specifically `Õ(√m)` for certain components, influencing their optimization landscape.
- Unfolded networks satisfy the PL* condition within a specific region of their parameter space, guaranteeing that gradient descent can achieve near-zero training loss in the over-parameterized regime.
- The model-aware, structured architecture of unfolded networks (LISTA and ADMM-CSNet) results in a larger minimum eigenvalue of the tangent kernel at initialization compared to standard FFNNs, indicating a more favorable optimization landscape.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.