Reproduce “An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiongtao Zhou, Jie He, Yuhua Ke, Guangyao Zhu, Víctor Gutiérrez-Basulto, Jeff Z. Pan

An empirical study systematically evaluates Parameter-Efficient Fine-Tuning (PEFT) methods for Multimodal Large Language Models (MLLMs) across varying data scales and architectural configurations. The research identifies the Adapter method as the most robust performer, consistently achieving superior accuracy, stability, and generalization while effectively mitigating hallucination rates in MLLMs.

Key insights

  • Fine-tuning connector layers alongside the LLM generally improves performance on unseen datasets, while freezing them often suffices or performs better on seen datasets.
  • The optimal placement of PEFT modules varies: LoRA and IA3 benefit from placement in both attention and MLP layers, whereas Adapter performs best when placed only in MLP layers.
  • Adapter consistently demonstrated the strongest generalization capabilities and effectively mitigated hallucinations, often reducing hallucination rates with further fine-tuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.