Reproduce “Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tomohiro Sawada, Kartik Goyal
The authors propose a LoRA fine-tuning penalty that encourages adapter updates to spread strength across more singular directions, rather than leaving configured rank underused. In mixed-domain supervised fine-tuning of Qwen2.5-0.5B at rank 896, the penalty raised mean effective rank from 360.537 to 667.960 and lowered training PPL from 4.773 to 4.381 and Wiki PPL from 19.601 to 15.290. These results cover one setup, not general retention.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.