Reproduce “SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bei Yan, Zhiyuan Chen, Yuecong Min, Jie Zhang, Jiahao Wang, Xiaozhen Wang, Shiguang Shan
SHALE is introduced as a scalable benchmark for fine-grained hallucination evaluation in Large Vision-Language Models (LVLMs), assessing both faithfulness and factuality under various clean and noisy input conditions. The research reveals that LVLMs particularly struggle with factual accuracy and relational understanding, with semantic perturbations causing the most significant performance degradation.
Key insights
- Generative tasks (FFQ, IC) are more challenging for LVLMs and more susceptible to performance degradation under noisy conditions compared to discriminative tasks (YNQ, MCQ).
- LVLMs exhibit greater difficulty with factuality hallucinations, especially in sports and politics domains, and with relational faithfulness aspects like size and spatial relationships.
- Semantic perturbations, such as scene text injection and misleading instruction prefixes, induce more substantial performance drops in LVLMs than low-level visual noise, highlighting sensitivity to linguistic cues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.