Reproduce “FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bowen Yan, Zhengsong Zhang, Liqiang Jing, Eftekhar Hossain, Xinya Du
FIHA introduces an automated and cost-effective framework for evaluating fine-grained hallucinations in large vision-language models (LVLMs), utilizing Davidson Scene Graphs (DSG) to model dependencies between questions. Its benchmark, FIHA-v1, demonstrates that even advanced LVLMs like GPT-4V struggle with attribute and relation hallucinations, with performance decreasing significantly when evaluated under dependency-aware conditions.
Key insights
- The Davidson Scene Graph (DSG) mechanism is critical for reliable hallucination assessment, revealing deeper model weaknesses by propagating errors from foundational object recognition to dependent attribute and relation questions.
- LVLMs, even top-performing ones, exhibit greater difficulty with attribute and relation hallucinations compared to object existence, a challenge exacerbated in out-of-distribution visual conditions like dense fog.
- Automated, LLM-free Q&A generation can achieve diversity comparable to LLM-based methods, while significantly reducing evaluation costs and improving scalability for benchmark creation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.