Reproduce “LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanan Li, Fanxu Meng, Muhan Zhang, Shiai Zhu, Shangguang Wang, Mengwei Xu
LoRASuite transfers LoRA adapters between successive versions of the same LLM architecture by combining embedding-based parameter transformations with CKA-guided layer alignment and Hungarian attention-head matching. With only 100 examples for lightweight refinement, it reached 43.80% average accuracy on MiniCPM mathematical tasks versus 22.82% for small-data LoRA, while reducing reported training time by 78.23% and memory use by about 5.5 GB relative to full retraining.
Key insights
- Representation similarity can provide useful structural correspondences for adapter transfer: CKA-based layer mapping and interaction-based head matching outperform simpler ordering and several alternative similarity methods in the reported ablations.
- Parameter transformation alone is generally insufficient; lightweight fine-tuning is needed to recover task performance and address numerical instability in the transferred updates.
- Transfer quality depends on the model upgrade and tuning configuration. LoRASuite can approach or exceed full-data LoRA for some upgrades, but remains substantially behind it for others such as Pythia, Bloom, and Llama on the evaluated math tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.