Reproduce “MERGETUNE: Continued Fine-Tuning of Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenqing Wang, Da Li, Xiatian Zhu, Josef Kittler
MERGETUNE, a method developed by researchers at the University of Surrey and Samsung AI Centre Cambridge, introduces "Continued Fine-Tuning" to recover pretrained knowledge in vision-language models after they have been fine-tuned, addressing catastrophic forgetting without requiring original pretraining data. The approach demonstrated consistent improvements in generalization, for instance, boosting CoOp's base-to-novel harmonic mean accuracy by +5.58% and outperforming state-of-the-art ensembling methods in robust generalization.
Key insights
- Linear Mode Connectivity (LMC) can be effectively adapted as a learning objective to actively integrate knowledge from distinct neural network states (zero-shot and fine-tuned) post-adaptation.
- A second-order Taylor expansion provides a tractable and replay-free surrogate for the original pretraining loss, enabling knowledge preservation without requiring the vast original datasets.
- A dual-anchored LMC objective, combining a standard downstream task loss with LMC constraints to both the zero-shot and fine-tuned models, regularizes training and facilitates robust knowledge recovery.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.