Reproduce “Chimera: Diagnosing Shortcut Learning in Visual-Language Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ziheng Chi, Yifan Hou, Chenxi Pang, Shaobo Cui, Mubashara Akhtar, Mrinmaya Sachan

A diagnostic test suite, CHIMERA, reveals that current open-source Visual-Language Models (VLMs) often rely on shortcut behaviors—visual-memorization, knowledge-recall, and particularly Clever-Hans—for diagram comprehension, rather than exhibiting genuine multimodal reasoning. Evaluations across 15 VLMs demonstrated significant reliance on linguistic priors and memorized knowledge, especially compared to their performance on fundamental visual entity recognition tasks.

Key insights

  • VLMs are heavily language-centric, achieving significantly higher accuracy with textual representations of diagram content compared to visual or semantic inputs.
  • Shortcut learning is widespread across VLMs, manifesting as visual-memorization, knowledge-recall, and especially Clever-Hans strategies.
  • Larger VLMs demonstrate some reduced reliance on Clever-Hans shortcuts in the presence of visual input, but still exploit linguistic priors when visual information is absent or insufficient.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.