Reproduce “Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kun Gu, De Cheng, Zhipeng Xu, Lingfeng He, Di Xu, Nannan Wang
M²LE addresses rehearsal-free, class-incremental learning by adapting LoRA updates across sequential tasks without retaining earlier examples. It replaces a shared merge scalar with element-wise strengths through HIM and trains updates with TSP to reduce sensitivity to later merges. HIM uses diagonal Fisher estimates under local quadratic approximations, while TSP simulates interference with current-task data. Without replay or classifier alignment, M²LE reaches 81.34% final accuracy on 10S-ImageNet-R versus 79.95% for LoRA-P&M, based on three-seed means.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.