Reproduce “Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shounak Datta, Dhanasekar Sundararaman
Researchers from Indian Statistical Institute and Duke University introduced Context-Aware Object Similarities (CAOS), a framework to evaluate object hallucination in Large Vision-Language Models (LVLMs) by measuring semantic relationships between hallucinated objects and ground-truth, sequential context, and frequent training objects. The framework revealed that LVLMs frequently hallucinate objects similar to those most common in their training data, and this tendency varies across models and is robust to prompt variations.
Key insights
- All evaluated LVLMs exhibit a strong tendency to hallucinate objects semantically similar to frequently occurring objects in their training datasets, as indicated by consistently higher CAOS_K scores.
- A trade-off exists among models where those with a higher influence from frequent training objects (high CAOS_K) generally show less semantic connection to ground-truth objects (CAOS_T) or preceding textual context (CAOS_X).
- The top-3 most frequent objects in training data disproportionately contribute to object hallucinations across various LVLMs, influencing both in-domain and out-of-domain hallucinations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.