Reproduce “MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shen Yuan, Yin Zheng, Taifeng Wang, Binbin Liu, Hongteng Xu
Researchers from Renmin University of China and ByteDance developed MoORE, an SVD-based method that transforms large language model layers into a Mixture of Orthogonal Rank-one Experts (MoORE) for multi-task adaptation. This approach improves conflict- and oblivion-resistance, outperforming existing PEFT and MoE-PEFT methods on multi-task benchmarks while maintaining competitive inference efficiency.
Key insights
- The orthogonality of SVD-derived experts prevents information redundancy and undesired interferences, directly mitigating task conflict in multi-task adaptation.
- Maintaining the column space of the original weight matrix via the SVD-based expert structure is critical for preserving pre-training ability and mitigating catastrophic forgetting.
- Combining task-level and sample-level routing allows for dynamic and efficient allocation of expert contributions, optimizing the model's response to diverse tasks and inputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.