Reproduce “Don't Blink: Evidence Collapse during Multimodal Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Suresh Raghu, Satwik Pandey
Vision-Language Models (VLMs) progressively lose visual grounding during extended reasoning, a phenomenon termed "evidence collapse." This study demonstrates this decay is universal across models, but its impact on correctness is highly task-conditional, leading to a "confident-but-blind" failure mode that is hazardous for tasks requiring sustained visual reference and benign for those transitioning to symbolic processing.
Key insights
- Visual attention to relevant image regions universally decays from the start of reasoning to the end of the answer in RVLMs, a phenomenon termed "evidence collapse."
- The implications of visual disengagement for VLM correctness are task-conditional; a "confident-but-blind" state is risky for sustained visual-reference tasks but benign for symbolic ones.
- Text-only uncertainty signals, while strong baselines, are insufficient to detect visual evidence collapse, necessitating modality-specific visual signals for robust VLM monitoring.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.