Reproduce “Effectively Enhancing Vision Language Large Models by Prompt Augmentation and Caption Utilization” on OpenResearch

About this paper

Minyi Zhao, Jie Wang, Zhaoyang Li, Jiyuan Zhang, Zhenbang Sun, Shuigeng Zhou

The PACU framework improves the robustness of Vision Language Large Models (VLLMs) against augmented or varied input prompts, mitigating hallucination by integrating prompt augmentation and image caption utilization during instruct-tuning. It yields up to a 6.9% increase in accuracy on augmented CIEM datasets for various VLLM architectures and enhances performance on 8 out of 13 general VLLM benchmarks.

Key insights

• VLLMs exhibit a critical vulnerability to augmented or perturbed prompts, leading to performance degradation and hallucination, a challenge not fully addressed by previous anti-hallucination methods.
• Training VLLMs with a diverse pool of automatically generated and quality-filtered augmented prompts significantly enhances their ability to process varied input formulations.
• Incorporating image captions as explicit prior knowledge during the response generation phase effectively guides the LLM, complementing visual features and reducing hallucination.

Summary by alphaXiv.