Reproduce “LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxuan Hu, Jing Zhang, Xiaodong Chen, Zhe Zhao, Cuiping Li, Hong Chen
LoRS introduces an efficient low-rank adaptation method tailored for sparse Large Language Models, which preserves the model's sparsity while significantly reducing memory and computational overhead. The approach effectively restores and improves the performance of pruned models on various downstream tasks.
Key insights
- Intermediate masked weight matrices in the computational graph can be recomputed on-the-fly during the backward pass to significantly reduce memory footprint with minimal additional computation.
- Rearranging matrix multiplications and selectively ignoring the mask in the backward pass can substantially reduce MACs required for gradient computation in sparse fine-tuning.
- Gradient-based adapter initialization, by mimicking initial full fine-tuning updates, leads to faster convergence and improved performance, especially on smaller datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.