Reproduce “Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenjin Hou, Wei Liu, Han Hu, Xiaoxiao Sun, Serena Yeung-Levy, Hehe Fan
VIA-Bench, a new benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on visual illusions and anomalies, reveals a substantial performance gap between MLLMs and human perception. The best MLLM achieved 69.23% accuracy compared to human 93.30%, and Chain-of-Thought reasoning often degrades MLLM performance in these challenging visual contexts.
Key insights
- Current MLLMs exhibit a substantial gap in perceptual and reasoning capabilities compared to humans when faced with visual illusions and anomalies.
- Chain-of-Thought (CoT) prompting frequently fails to enhance, and often degrades, MLLM performance in these challenging visual contexts due to 'hallucination reinforcement'.
- MLLMs' failures often stem from fundamental visual misperceptions, leading to high uncertainty and 'overthinking' phenomena where models struggle to resolve visual ambiguity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.