Reproduce “Towards Artwork Explanation in Large-scale Vision Language Models” on OpenResearch

About this paper

Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe

Researchers at Nara Institute of Science and Technology introduce the ExpArt dataset and specialized metrics for a novel artwork explanation generation task, designed to evaluate how large-scale vision-language models integrate complex knowledge. The study reveals a significant reliance on textual cues for comprehensive explanations and finds that fine-tuning substantially enhances performance, particularly for Qwen-VL-Chat.

Key insights

• Large-scale Vision-Language Models (LVLMs) heavily rely on textual cues like artwork titles for generating comprehensive explanations, indicating difficulties in deriving knowledge solely from visual inputs.
• The integration of vision and language can degrade artistic knowledge from base Large Language Models (LLMs) in most LVLMs, though Qwen-VL-Chat uniquely demonstrated enhanced performance in combined settings.
• Fine-tuning models on the ExpArt dataset is crucial for achieving high performance in artwork explanation, significantly surpassing few-shot learning by enabling the acquisition of domain-specific vocabulary and explanation structures.

Summary by alphaXiv.