Reproduce “LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Juzheng Zhang, Jiacheng You, Ashwinee Panda, Tom Goldstein
Researchers at the University of Maryland and Tsinghua University introduce LoRI, a parameter-efficient fine-tuning method that reduces trainable parameters by up to 95% compared to LoRA. LoRI maintains or improves task performance while significantly reducing cross-task interference during adapter merging and mitigating catastrophic forgetting in continual learning.
Key insights
- Freezing the A matrices in LoRA adapters (LoRI-D) can yield comparable or superior performance to training them, indicating that effective adaptation doesn't necessitate updating these projections.
- A high degree of sparsity (e.g., 90%) can be introduced into the B matrices (LoRI-S) without significant performance loss, revealing substantial parameter redundancy and focusing updates on critical parameters.
- The approximate orthogonality of LoRI adapters in distinct subspaces, stemming from their fixed random A matrices, naturally minimizes interference when merging multiple adapters for multi-task learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.