Reproduce “MYOPIA: Understanding the Limits of Tree-Based Reasoning in Vision-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Islam Mesabah, Sary Elmansoury, Gerrit Großmann, Peter Neigel, Sebastian J. Vollmer

The authors introduce MYOPIA to test visual classification through branching questions. An LLM proposes label-splitting questions, humans refine the tree, and a vision-language model answers each node in a separate call. Tree traversal scored below zero-shot prompting in all 12 comparisons; on CIFAR-10, GPT-4o scored 76.00% versus 87.90%. This result concerns the tested setup, not hierarchical inference generally.

Key insights

  • For the evaluated trees, one mistaken answer can eliminate the correct class with no opportunity to recover; question-level accuracy therefore does not guarantee strong end-to-end classification.
  • Models answered tree questions more accurately when given a class label than when required to identify the relevant feature in an image; strict answer extraction also left some traversals unresolved.
  • Results depend on prompt wording: zero-shot accuracy varied substantially across paraphrases, so the primary comparison should not be treated as invariant to prompt choice.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.