Reproduce “OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yifeng Xiong, Xiaohui Xie
OPLoRA, a parameter-efficient fine-tuning (PEFT) method, mathematically guarantees the preservation of a large language model's dominant pre-trained knowledge by applying double-sided orthogonal projections to LoRA updates. This approach consistently demonstrates superior knowledge retention on held-out tasks while maintaining competitive task-specific performance across commonsense, mathematical reasoning, and code generation benchmarks for LLaMA-2 7B and Qwen2.5 7B models.
Key insights
- Essential pre-trained knowledge in large language models is concentrated within the dominant singular directions of their weight matrices.
- Catastrophic forgetting in LoRA largely occurs because learned updates interfere with these dominant singular subspaces, overwriting critical prior knowledge.
- Constraining LoRA updates to the orthogonal complement of the dominant singular subspaces effectively prevents interference and thereby preserves pre-trained knowledge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.