Reproduce “MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiale Li, Mingrui Wu, Zixiang Jin, Hao Chen, Jiayi Ji, Xiaoshuai Sun, Liujuan Cao, Rongrong Ji
Introduces MIHBench, the first benchmark for systematically evaluating object-related hallucinations in Multimodal Large Language Models (MLLMs) across multiple images, and proposes Dynamic Attention Balancing (DAB), a training-free mitigation mechanism. The research reveals that hallucinations increase with more input images and that DAB consistently improves MLLM performance on multi-image tasks.
Key insights
- The likelihood of MLLM hallucination progressively increases with the number of input images, indicating challenges in global semantic integration across diverse visual contexts.
- Single-image hallucinations strongly correlate with multi-image hallucinations, often propagating and amplifying in more complex multi-image reasoning scenarios (Pearson coefficient of 0.6153).
- MLLMs exhibit attention biases, tending to overlook inconsistencies in later-positioned negative samples and showing a bias towards assuming consistency when positive examples are abundant.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.