Reproduce “Position: Reasoning After Perception Means Reasoning Without Vision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongcheng Gao, Zihao Huang, Jingyi Tang, Lin Xu, Xinhao Li, Haoyang Li, Yue Liu, Minhua Lin and 7 more
Researchers developed the Turing Eye Test (TET) to probe fine-grained visual reasoning in Multimodal Large Language Models (MLLMs), revealing that 15 state-of-the-art MLLMs achieve near-zero performance on visually precise tasks. The work identifies the vision encoder and its subsequent projection as the primary bottleneck, causing irreversible information collapse that language models cannot recover.
Key insights
- State-of-the-art MLLMs exhibit "collective blindness" on fine-grained visual tasks, with inference-time interventions like in-context learning showing negligible improvement.
- The vision encoder acts as the primary bottleneck, causing irreversible loss of spatial and geometric information during its initial encoding and projection.
- MLLMs display a "semantic bias," prioritizing coarse semantic features over fine-grained visual details, as evidenced by paradoxical accuracy increases with image downsampling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.