Reproduce “CoLA: Collaborative Low-Rank Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiyun Zhou, Chang Yao, Jingyuan Chen
CoLA (Collaborative Low-Rank Adaptation) is a flexible parameter-efficient fine-tuning framework for large language models that addresses limitations of existing LoRA variants through an improved architecture, an extended PiSSA initialization, and collaborative matrix interaction strategies. Experiments demonstrate CoLA's consistent outperformance of baseline methods across single and multi-domain tasks, particularly showing effectiveness in low-data scenarios.
Key insights
- CoLA demonstrates versatility across single and multi-domain tasks, consistently performing well.
- Extended PiSSA initialization significantly improves CoLA's performance, especially in low-data scenarios, compared to standard LoRA.
- Optimal performance is consistently achieved when the number of A matrices is less than the number of B matrices (#A < #B), suggesting B matrices capture more task-specific details.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.