Reproduce “PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assembly” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Liang Ma, Jiajun Wen, Min Lin, Rongtao Xu, Xiwen Liang, Bingqian Lin, Jun Ma, Yongxin Wang and 7 more
PhyBlock introduces a progressive benchmark to evaluate Vision-Language Models' physical understanding and multi-step planning capabilities through 3D block assembly tasks, revealing that state-of-the-art models exhibit a steep performance drop with increasing task complexity and struggle with spatial orientation and dependency reasoning. It comprehensively evaluates 23 leading VLMs and establishes a significant gap between current AI and human performance.
Key insights
- VLMs exhibit a steep performance drop, often by more than half, from simple to complex assembly tasks, indicating limitations in long-horizon planning and dependency reasoning.
- A significant perception-reasoning gap exists; models excel at identifying object properties but struggle dramatically with VQA questions requiring dynamic reasoning or causal inference.
- Universal error modes across all VLM architectures include incorrect spatial orientation and failure to meet fundamental support dependencies during assembly.
- Enabling 'thinking mode' or chain-of-thought prompting provides minimal to no performance improvement for physical understanding and planning tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.