Reproduce “VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation” on OpenResearch

About this paper

Seongheon Park, Changdae Oh, Hyeong Kyu Choi, Sean Du, Sharon Li

A training-free framework, VAUQ, quantifies a Large Vision-Language Model's reliance on visual evidence for self-evaluation, addressing hallucinations by explicitly measuring visual grounding. It consistently outperforms existing baselines in accuracy across diverse LVLMs and datasets, achieving up to +21.4% AUROC improvement and a 94.6% reduction in inference time compared to previous state-of-the-art methods.

Key insights

• Quantifying the reduction in predictive uncertainty due to visual input is an effective proxy for measuring an LVLM's reliance on visual evidence and its visual grounding.
• Unsupervised identification and masking of task-relevant "core regions" through aggregated visual attention from intermediate transformer layers effectively captures semantically important visual information.
• Combining predictive entropy (effective for factual accuracy) with vision-aware uncertainty (critical for counterfactual scenarios) provides a robust and balanced self-evaluation signal across diverse contexts.

Summary by alphaXiv.