Reproduce “Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Davide Bucciarelli, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Researchers at the University of Modena and Reggio Emilia conducted an experimental analysis showing that Multimodal Large Language Models (MLLMs) require fine-tuning to align with the style of image captioning benchmarks. Prompt learning proved to be a computationally efficient parameter-efficient fine-tuning (PEFT) strategy, improving in-domain performance while preserving generalization across diverse out-of-domain datasets.
Key insights
- Zero-shot MLLMs produce broadly descriptive content with high visual-textual similarity (CLIP-Score) but often struggle to match the concise, factual style of standard image captioning benchmarks, frequently introducing hallucinations.
- Full fine-tuning significantly boosts in-domain image captioning performance but can lead to a reduction in generalization ability when tested on out-of-domain datasets.
- Parameter-Efficient Fine-tuning (PEFT) methods, particularly prompt learning, offer a balanced approach by enhancing in-domain performance while maintaining robust generalization across various out-of-domain tasks with minimal computational cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.