Reproduce “Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeping Yu, Sophia Ananiadou
A training-free framework, Locate-then-Merge, with its Neuron-Fusion strategy, mitigates catastrophic forgetting in multimodal large language models (MLLMs) by selectively restoring neuron-level parameter changes. The method achieves improved overall performance across language and visual benchmarks for Llama3 and Mistral-based MLLMs, while also reducing common generation errors like "Not-Known" and "Context-Hallucination" problems.
Key insights
- Catastrophic forgetting in MLLMs can be effectively addressed by focusing on neuron-level parameter changes, where a few neurons undergo large shifts for new abilities, while widespread small changes degrade old ones.
- The Locate-then-Merge framework offers a unifying and more targeted approach to model merging, moving beyond indiscriminate parameter-level operations.
- Restoring the influence of a select group of neurons with large parameter shifts is crucial for preserving visual abilities, while suppressing other minor parameter changes helps recover language proficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.