Reproduce “LoRA Training in the NTK Regime has No Spurious Local Minima” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Uijeong Jang, Jason D. Lee, Ernest K. Ryu
This work theoretically analyzes Low-Rank Adaptation (LoRA) within the Neural Tangent Kernel (NTK) regime, establishing that the optimization landscape of LoRA fine-tuning is free of spurious local minima under certain conditions and that the resulting solutions generalize well. It demonstrates that optimal weight updates in this regime are inherently low-rank and provides quantitative guarantees for trainability and generalization.
Key insights
- Even full fine-tuning in the NTK regime yields low-rank optimal weight updates, specifically with rank $r$ such that $r(r+1)/2 \le KN$, where $K$ is output dimension and $N$ is data points.
- For LoRA ranks $r$ satisfying $r(r+1)/2 > KN$, a slightly perturbed loss function has no spurious local minima, ensuring gradient-based optimizers converge to global optima.
- LoRA solutions found under these conditions possess strong generalization guarantees, bounded by factors including the nuclear norm of the optimal solution and data size.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.