Reproduce “A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haonan He, Jingqi Ye, Minglei Li, Zhengbo Wang, Tao Chen, Lei Bai, Peng Ye
A unified study consolidates over 20 Low-Rank Adaptation (LoRA) variants, providing a comprehensive taxonomy, theoretical review, and the `LoRAFactory` codebase for standardized evaluation. Extensive experiments across diverse domains reveal vanilla LoRA's strong competitive performance when optimally tuned, often matching or surpassing many of its more complex variants, and highlight the critical sensitivity of all variants to learning rate.
Key insights
- LoRA and its variants demonstrate a highly pronounced sensitivity to learning rate, often more so than to other hyperparameters, necessitating extensive sweeps for fair comparative evaluation.
- With proper hyperparameter tuning, vanilla LoRA consistently matches or surpasses the performance of most of its proposed variants across diverse tasks, challenging prior claims of variant superiority.
- Architectural complexities and enhanced operational mechanisms in LoRA variants frequently incur significant computational and memory overheads, highlighting a trade-off with efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.