Reproduce “IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang and 1 more

IconQA presents a new benchmark and dataset for evaluating AI systems' ability to understand abstract diagrams and perform complex visual language reasoning, drawing from elementary math word problems. The work reveals a significant performance gap between human and state-of-the-art AI models, while demonstrating that a proposed Patch-TRM baseline, incorporating icon-specific pre-training, surpasses existing methods.

Key insights

  • Abstract diagrams based on elementary math word problems provide a robust and accessible domain for evaluating comprehensive cognitive reasoning in AI, avoiding reliance on complex domain expertise.
  • Pre-training vision encoders on domain-specific abstract icon datasets, rather than natural images, is crucial for achieving higher performance in understanding and reasoning about abstract visual information.
  • Current state-of-the-art VQA models have a substantial performance gap (10-20% difference) compared to human accuracy in abstract visual language reasoning, indicating a need for more advanced AI architectures.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.