Reproduce “Aligning Modalities in Vision Large Language Models via Preference Fine-tuning” on OpenResearch

About this paper

Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, Huaxiu Yao

This paper introduces POVID (Preference Optimization in VLLM with AI-Generated Dispreferences), a method for fine-tuning Vision Large Language Models to mitigate hallucinations and improve modality alignment. POVID achieves this by automatically generating preference data with AI-crafted hallucinatory responses and responses to noisy images, reducing hallucinations by an average of 31.78% across benchmarks and enhancing overall VLLM performance.

Key insights

• AI-generated dispreferences, contrasting ground truth with intelligently crafted hallucinations, are highly effective and scalable for improving VLLM modality alignment without human annotation.
• Combining two distinct strategies for generating dispreferences—deliberately hallucinated text and responses to noisy images—tackles both explicit and implicit hallucination tendencies of VLLMs.
• Preference tuning specifically designed to address modality misalignment by leveraging ground truth as preferred responses leads to VLLMs that re-prioritize visual attention over textual biases, improving grounding.

Summary by alphaXiv.