Reproduce “ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yilang Zhang, Xiaodong Yang, Yiwei Cai, Georgios B. Giannakis

ScaLoRA introduces an efficient low-rank adaptation method for fine-tuning large language models, dynamically achieving high-rank weight updates by optimally scaling and merging low-rank increments. This approach, developed by researchers from Morgan Stanley, Visa Research, and the University of Minnesota, addresses performance and convergence limitations of traditional LoRA, demonstrating over 0.5% average performance gain on GLUE and outperforming LoRA (r=32) with a lower rank (r=8) on commonsense reasoning tasks.

Key insights

  • Directly identifying optimal low-rank adapters to minimize the loss function's quadratic upper bound would require computationally prohibitive Singular Value Decomposition (SVD) of the full gradient at each step.
  • By constraining new adapters to be scaled versions of existing ones, ScaLoRA provides analytical solutions for optimal scalar and column-wise scaling factors, ensuring efficient updates for adaptive optimizer moment estimators without re-initialization.
  • The intermittent variant, ScaLoRA-I, performs optimal scaling and merging operations only every 'I' iterations, amortizing the per-step computational cost and enhancing scalability while largely preserving performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.