Reproduce “CutPaste&Find: Efficient Multimodal Hallucination Detector with Visual-aid Knowledge Base” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cong-Duy Nguyen, Xiaobao Wu, Duc Anh Vu, Shuai Zhao, Thong Nguyen, Anh Tuan Luu
Researchers from Nanyang Technological University introduce CutPaste&Find, an efficient, training-free framework for detecting object, attribute, and relationship hallucinations in Large Vision-Language Model outputs. This offline solution leverages off-the-shelf modules and a Visual-aid Knowledge Base, achieving higher accuracy than API-dependent methods on POPE and R-Bench benchmarks.
Key insights
- Off-the-shelf visual and linguistic modules, combined with scene graph parsing, can efficiently detect hallucinations without relying on costly, iterative LVLM inference.
- A Visual-aid Knowledge Base, constructed from richly annotated datasets like Visual Genome, offers a reliable source of visual prior knowledge for verifying claims.
- Introducing a scaling factor based on prior knowledge from the VaKB enhances the robustness of visual-linguistic similarity scores, leading to more accurate hallucination identification.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.