Reproduce “Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhengbao He, Ruiqi Ding, Zhehao Huang, Ruikai Yang, Tao Li, Xiaolin Huang
Compress-then-Merge (CtM) consolidates multiple LoRA adapters into a single target-rank update by learning shared left and right subspaces before task updates are merged. Across eight vision tasks and six language tasks, it improves average normalized performance over merge-then-compress baselines, while retaining task information more evenly and reducing LLaMA3-8B subspace-optimization time from 690.83 to 20.77 seconds through CoreSpace acceleration.
Key insights
- Constraining the subspace before merging avoids the information loss caused by applying SVD only after unrestricted task updates have interacted.
- CtM distributes rank capacity more evenly: on vision tasks, mean energy retention rises from 69.70 to 87.94, worst-task retention from 41.94 to 85.24, and retention variability falls substantially.
- Reweighting helps but does not fully explain the gains; learned shared subspaces and coordinate-space merging are important contributors to the improvement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.