Reproduce “From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianxu Wang, Zhuofan Zhang, Ziyu Zhu, Yue Fan, Jing Xiong, Pengxiang Li, Xiaojian Ma, Qing Li

Anywhere3D-Bench introduces a holistic benchmark for multi-level visual grounding in 3D scenes, covering area, space, object, and part granularities. This evaluation framework uncovers a substantial gap between current models and human capabilities in complex 3D spatial reasoning, particularly for unoccupied spaces and fine-grained object parts.

Key insights

  • A significant performance gap exists between human capabilities (95% accuracy) and the best-performing models (around 46% overall) in multi-level 3D visual grounding.
  • Space-level and part-level visual grounding are the most challenging tasks for all evaluated models, highlighting deficiencies in current AI's capacity for complex spatial and fine-grained reasoning.
  • Models equipped with 'thinking' (chain-of-thought) capabilities consistently outperform their non-thinking counterparts, and MLLMs generally surpass specialized 3D models due to broader pretraining, although MLLMs still show limited gains in leveraging 2D inputs for 3D spatial relational reasoning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.