Reproduce “RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoyu Wang, Tianci Liu, Ruirui Li, Monica Cheng, Tuo Zhao, Jing Gao
RoseLoRA introduces a parameter-efficient fine-tuning method that enforces row and column-wise sparsity on low-rank adaptation matrices, enabling sparse updates to pre-trained language models. This approach improves precision in knowledge editing, reduces catastrophic forgetting, and achieves strong performance across various fine-tuning tasks with significantly fewer trainable parameters.
Key insights
- A theoretical proposition and lemma demonstrate that constraining row and column sparsity in low-rank matrices (A and B) effectively translates to a guaranteed level of sparsity in their combined product (BA).
- Inducing sparsity directly at the product level of low-rank update matrices enables more surgical and precise parameter modifications, which is crucial for targeted knowledge editing without broad side effects.
- Highly efficient and effective adaptation of large language models, even for complex tasks like instruction following, can be achieved by updating an extremely sparse subset of parameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.