Reproduce “MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yang Shi, Yifeng Xie, Minzhe Guo, Liangsi Lu, Mingxuan Huang, Jingchao Wang, Zhihong Zhu, Boyan Xu and 1 more
MMErroR introduces a benchmark for evaluating Vision-Language Models' ability to diagnose erroneous reasoning by requiring them to detect and classify specific error types within multi-modal reasoning chains. Experiments on state-of-the-art models reveal they significantly lag human performance in this diagnostic task, indicating a gap in their genuine multi-modal understanding.
Key insights
- A VLM's ability to accurately diagnose error types in reasoning chains correlates strongly with its performance on original Visual Question Answering (VQA) tasks, suggesting error diagnosis is linked to fundamental comprehension.
- Robust multi-modal semantic alignment, observed via logit lens visualization, is critical for successful error detection and diagnosis, with failures often linked to disrupted cross-modal mapping.
- Providing models with explicit error type information significantly improves their ability to correct VQA answers, indicating that diagnostic insight offers actionable guidance for improving model outputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.