Reproduce “HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen and 1 more
Researchers from City University of Hong Kong and collaborators introduced HumanEval-V, a benchmark evaluating Large Multimodal Models' high-level visual reasoning over indispensable, complex diagrams in coding tasks. Top models like Claude 3.5 Sonnet achieved only 36.8% pass@1, indicating current LMMs struggle significantly with interpreting spatial transformations and topological relationships in visual information.
Key insights
- HumanEval-V poses a substantial challenge for current LMMs, revealing significant limitations in genuine diagram understanding beyond superficial perception.
- Decoupling visual understanding (generating text descriptions) from code generation (V2T2C w/ SC) often improves performance, indicating stronger vision-to-language alignment than vision-to-code in current LMMs.
- LMMs primarily struggle with core visual reasoning components such as spatial transformations, topological relationships, and dynamic patterns, rather than their coding proficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.