Reproduce “Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Heekyung Lee, Jiaxin Ge, Tsung-Han Wu, Minwoo Kang, Trevor Darrell, David M. Chan
A new hand-generated benchmark of 432 diverse English-language rebus puzzles evaluates Vision-Language Models' abstract and lateral reasoning capabilities. The study reveals that even advanced VLMs significantly lag human performance, particularly in tasks involving negation or visual metaphors, and found that direct visual input is critical for their reasoning processes.
Key insights
- VLMs exhibit an "awareness vs. execution gap," where they may recognize the required reasoning type but lack the fundamental capacity to execute it effectively, as observed from minimal gains with skill-guided prompts.
- Direct visual access is crucial for reasoning models; removing visual input (caption-only experiments) significantly degrades performance, suggesting an iterative examination of visual content during complex reasoning.
- VLM reasoning capabilities are stronger for rule-based symbolic transformations but notably weaker for abstract concepts like negation, visual metaphors, and nuanced phonetic manipulation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.