Reproduce “DASH: Detection and Assessment of Systematic Hallucinations of VLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Maximilian Augustin, Yannic Neuhaus, Matthias Hein

Maximilian Augustin and Yannic Neuhaus from the Tübingen AI Center developed DASH, an automatic pipeline to identify and analyze systematic hallucination patterns in Vision-Language Models in open-world settings. The pipeline successfully revealed consistent VLM hallucination patterns and demonstrated that targeted fine-tuning with hallucination-inducing images can reduce these errors without performance loss.

Key insights

  • VLM hallucinations are often systematic and can transfer across different model architectures, indicating common underlying vulnerabilities.
  • The LLM backbone and vision encoder components of a VLM play a more significant role in its susceptibility to hallucinations than overall model size.
  • Consistent error patterns exist, including confusing visually similar objects, hallucinating contextually associated objects, and over-generalizing from common co-occurrences.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.