Reproduce “The Impact of Initialization on LoRA Finetuning Dynamics” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Soufiane Hayou, Nikhil Ghosh, Bin Yu

The paper demonstrates that two common initialization schemes for Low-Rank Adaptation (LoRA) - Init[A] (random A, zero B) and Init[B] (zero A, random B) - lead to fundamentally different finetuning dynamics. Init[A] consistently achieves superior performance across various tasks and models, such as 90.69% accuracy on MNLI with RoBERTa-large compared to 89.47% for Init[B], by enabling larger learning rates and more efficient feature learning, despite internal instability in intermediate features.

Key insights

  • Despite both schemes yielding a zero initial LoRA output, Init[A] and Init[B] lead to fundamentally different finetuning dynamics and optimal learning rate scalings in the infinite-width limit.
  • Init[A] permits asymptotically larger stable learning rates (scaling as "Theta(n^(-1/2))") and achieves efficient feature learning where both A and B contribute significantly, although it exhibits "internal instability" where intermediate features (AZ) can grow as "Theta(n^(1/2))".
  • Init[B] requires smaller stable learning rates (scaling as "Theta(n^-1)") and, while preventing internal instability, leads to suboptimal feature learning where the B matrix contribution to updates vanishes in the large-width limit.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.