Reproduce “GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuan Feng, Yue Yang, Xiaohan He, Jiatong Zhao, Jianlong Chen, Zijun Chen, Daocheng Fu, Qi Liu and 3 more
Researchers developed GeoBench, a hierarchical benchmark with formally verified synthetic data, to diagnostically evaluate multimodal large language models (MLLMs) on geometric reasoning. The framework revealed that MLLMs struggle significantly with self-reflective reasoning, while capacities for sub-goal decomposition and irrelevant premise filtering are strong determinants of problem-solving success.
Key insights
- MLLMs' performance consistently declines with increasing task complexity, with self-reflective backtracking presenting the most significant challenge.
- Sub-Goal Decomposition and Irrelevant Premise Filtering are identified as critical reasoning bottlenecks, exhibiting the strongest correlations with overall geometric problem-solving success.
- Chain-of-Thought (CoT) prompting does not universally improve performance and can even degrade results in tasks like faulty branch localization, challenging its presumed efficacy in all complex reasoning scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.