Reproduce “Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guangzong Si, Hao Yin, Xianfei Li, Qing Ding, Wenlong Liao, Tao He, Pai Peng
Research from USTC and CowaRobot re-evaluates object hallucination in Multimodal Large Language Models, demonstrating distinct underlying causes for omission and fabrication errors. They introduce Visual Potential Field Calibration (VPFC), a training-free method that effectively reduces omission hallucinations without increasing fabrication errors, outperforming existing approaches on various benchmarks like POPE and CHAIR.
Key insights
- Omission hallucinations stem from MLLMs having low confidence in mapping correctly perceived visual features to linguistic expressions, not solely from a failure to recognize the object visually.
- Fabrication hallucinations arise from spurious cross-modal associations within the representation space, often due to statistical biases in training data, leading to high-confidence erroneous object inferences.
- Existing mitigation methods implicitly boost general visual attention, inadvertently increasing both desired omission reduction and undesired fabrication increases due to their unified cause assumption.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.