Reproduce “MERGETUNE: Continued Fine-Tuning of Vision-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenqing Wang, Da Li, Xiatian Zhu, Josef Kittler

MERGETUNE, a method developed by researchers at the University of Surrey and Samsung AI Centre Cambridge, introduces "Continued Fine-Tuning" to recover pretrained knowledge in vision-language models after they have been fine-tuned, addressing catastrophic forgetting without requiring original pretraining data. The approach demonstrated consistent improvements in generalization, for instance, boosting CoOp's base-to-novel harmonic mean accuracy by +5.58% and outperforming state-of-the-art ensembling methods in robust generalization.

Key insights

  • Linear Mode Connectivity (LMC) can be effectively adapted as a learning objective to actively integrate knowledge from distinct neural network states (zero-shot and fine-tuned) post-adaptation.
  • A second-order Taylor expansion provides a tractable and replay-free surrogate for the original pretraining loss, enabling knowledge preservation without requiring the vast original datasets.
  • A dual-anchored LMC objective, combining a standard downstream task loss with LMC constraints to both the zero-shot and fine-tuned models, regularizes training and facilitates robust knowledge recovery.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.