Reproduce “LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)” on OpenResearch

About this paper

Junsu Kim, Jaeyeon Kim, Ernest K. Ryu

This research provides theoretical guarantees for Low-Rank Adaptation (LoRA) training, proving that its optimization converges either to a low-rank global minimum or to a distinguishable spurious local minimum characterized by high rank and large magnitude. It demonstrates how standard practices like zero-initialization and weight decay implicitly steer the training process toward optimal, low-rank solutions.

Key insights

• In a 'special regime' where the loss landscape is very well-conditioned, every SOSP of the LoRA objective corresponds to a global minimum of the nuclear-norm regularized full fine-tuning loss.
• In a more general 'generic regime,' an SOSP is either a global minimum or a 'spurious local minimum' that is qualitatively distinct, characterized by having the maximal LoRA rank and a significantly larger magnitude.
• Zero-initialization and weight decay act as implicit biases, steering LoRA training towards low-rank, small-magnitude global minima and away from the high-rank, large-magnitude spurious local minima.

Summary by alphaXiv.